Don't hotplug a PS2 port - you risk burning out the port that way. It's okay to hotplug USB but not PS2.
From the look of search results, I think that board uses the Intel 440GX chipset. Correct me if I'm wrong.
It's basically a 440BX with double the RAM capacity.
I don't recall changing the CMOS setting but maybe I did many, many years ago and forgot about it.
I'm not sure if it was already mentioned, but make sure the CMOS battery is good. Some boards can get goofy when the battery is low. If the battery is good then reset CMOS. If you're doing that by removing the battery then remember it also needs to be unplugged.
I think the computer is a Pentium II and I don't know the speed but maybe 250 or 300 if that sounds about right.
It needs to be a 66FSB or 100FSB CPU, make sure it's not 133FSB. To avoid any possible incompatibility, the best bet is a Pentium 2, not a P3.
If I understood correctly it sounds like you're using 1 CPU. Does your board allow having an empty CPU slot? Some boards require both slots to be filled, either with a CPU or with a slot-1 terminator card. My Intel L440GX+ is strict about that. Other boards don't seem to care, but I don't know what Supermicro is like.
The memory is only 1 stick and should be 512mb, PC100 non ECC module. I think I can go up to 1gb of memory or more. There is 4 slots but I may have to keep it in pairs.
Do you have another system that can run memtest86 on that module?
The 440GX doesn't care about pairs, single modules are fine. 512MB modules are the max the chipset will run, but some boards might be picky about which ones they'll work with. I have an HP with that chipset that, if given a 512MB module it doesn't like, will start to go through the POST procedure but then freezes with a cryptic error code. With the HP I think it's a BIOS issue (the RAM is compatible on another 440GX board). I can tell by looking that your module is one my HP wouldn't like. It's more tolerant with 256MB and smaller modules.
Does your board require Registered memory? Does it require ECC? Some 440GX boards could have such requirements. As you mentioned, the module in your picture appears to be a basic unbuffered non-ECC module.
Do you have a smaller memory module you can try? Do you have any Registered memory you can try?
If you want to fill all those memory slots in the future, Intel would tell you they have to be Registered. Using only 1 slot, it probably just depends on the board manufacturer.
I think the most surely compatible RAM for a board like this would be Registered ECC PC100 256MB modules with 18 RAM ICs on them (9 per side) plus the smaller buffering chips underneath them.
512MB modules with 18 RAM ICs *should* work with this chipset, but my HP will only accept the ones that have 36 RAM ICs for whatever reason. In any case, 512MB is bleeding edge for this chipset and may tempt it to get cranky.
Do you have a multimeter? Can you try measuring all the supply voltages while the board is powered on? 5V and 12V are on the molex connectors, 3.3V (orange) and 5Vstandby (purple) can be backprobed from the ATX connector. Watch them to see if they're sagging while it tries to boot.
You can also check CPU Vcore at the metal tabs on the MOSFETs next to the CPU slots. Put the black probe against the PSU casing, red probe on the large metal tab of one of those MOSFETs. One will probably show you the 5V supply, the other will show you Vcore.
Marginal voltages could explain erratic behavior. Swapping the PSU doesn't really rule this out because you still don't know what voltages you were getting.
I have some Fortron FSP300-??? model PSUs. I use them all the time because they perform very well, but every one of them had bad caps inside. Many motherboards will shrug this off for a long time, but it certainly doesn't help. If you can replace caps then you might want to recap your PSU.
Firstly though, go through that list PCHP posted. I especially agree it looks like 2 pins on your parallel port are shorted together.
If that cap he mentioned is bulging (at all, even a tiny bit) then it's bad and might explain quirkiness with the keyboard/etc.