When building electronic projects, it's sometimes nice to have a PC around - for looking up datasheets online, pulling up schematics, or even for providing I/O while testing a circuit. My recommendation for a machine would be to use an older computer with a Pentium II 233 or so processor and as much RAM as you can cram into it. The older machine is nice in that it is fast enough for most simple things that it will be used for on a bench, but old enough that it can support all sorts of old cards and should have most of the "legacy" ports built into it.
These "legacy" (I hate that word - these are such good ports, they're just old - call them classics or something) ports are the standard parallel, serial, and MIDI. The parallel port, which long-time readers of this blog should know is my favorite, provides 12 digital output lines and 5 digital input lines. The serial port is another handy one which provides a TX (transmit) and RX (receive) line which can be used for programming, data-acquisition, and other similar uses. The unsung hero in my opinion of these classics is the MIDI port which provides 4 analog inputs which are each converted to an 8-bit value and 4 digital inputs.
In addition to ensuring that these ports are present, the machine should have a network interface card (NIC) if you are planning to get online, or at the least a modem and a phone line. It might also be wise to look for some of the older ISA cards that are available that provide extra ports and connectivity - check eBay, local auctions, or old computers that people might be willing to part with for these.
For software, I would suggest running either Linux or a version of Windows prior to Windows 2000 - Windows 98se is a good choice as it is the latest version before 2000 that isn't ME (I would advise no one to run ME...). The old versions are suggested because the versions of Windows that are NT based have security to prevent low-level access to the hardware ports. Also - it's nice to run real DOS sometimes for older applications and command-line based programs that you might write. Linux is also a very nice choice since it is so open, easy to program for, and lots of people who do hardware work run it.