Hrmmm...
I'd say stick with Windows 98SE because I can't really think of any compatibility you are losing over Windows 95. Both can be restarted in DOS mode, and, if all else fails, you can set up a DOS partition.
Mobo and CPU, I've been looking at a lot of different sockets lately, after restoring my Slot 1 workstation. I've found this (starting from newest):
--Socket A (aka 462) (AMD) seems to be where it's at, but you will pay, as the higher-end boards still demand a very high price. The older AMD CPUs from the late 90's to Early 2000's are solid, fun, cheap overclockers. The boards generally have a lot of PCI slots and many have legacy ISA, and they are easy to find in uATX if you want something little.
--Socket 478 (Intel) is a very similar ecosystem to Socket A, with a lot of the same amenities, but more stuff out there. Probably less expensive.
--Slots: AMD has Slot A, while Intel has Slot 1. For a true period piece, these are going to be it. You're going to see more ISA support and less integrated audio/graphics. Also a cheaper option than the above.
--Socket 7: Now we're getting back there, as socket 7 came about in 1995. This is where most of your AMD K5's and middle-aged Intel Pentium processors are slotted in, among others. Lots of interesting choices here, albeit slower than any of the above. Socket 7 processors are dime-a-dozen, so you might as well grab some extras from different brands just to play with. I wouldn't really recommend it for what you want to do; not powerful enough for later software.
--Socket 2-6: This is if you want to get really specific. For what you're doing, going with any of these will just create undue parts-chasing. I think my first computer used Socket 5.
--Socket 1: For 486's. Really not appropriate for what you are doing. BUT, if you enjoy this project, I do recommend reading about the 486 and building one for a pure DOS/Windows 3.1 machine.
Arrgh I forgot about socket 370; kinda the socket version of slot 1. Don't know much about it because I don't really care much about it, but it is worth looking into. Maybe relevant to your interests.
That said, I'd say Slot A would be kinda ideal. The AMD processors used are fully unlocked, so you can overclock and play around with them all you want, and they are cheap. Boards are pretty cheap, too.
When you've picked your socket or slot, it's all about what you can find. Try to get something that doesn't use a Via chipset. They aren't bad, but they aren't the best.
GPU: I would go ahead and get something more powerful. A Radeon 7xxx or higher will do everything, as will a GeForce2, and you'll only be paying maybe $10-$15 USD for it. 3DFX cards are collectors items and generally overpriced. The TNT cards are great, but I don't think they will get you those absolute ultra settings on the way later games. Sound card is going to have more impact on configuring for compatibility.
Sound card: an authentic Sound Blaster 16 ISA card is your best bet. Works with everything. For what you're doing, it would be silly not to grab one of these. Second Choice would be a SB AWE. Avoid Gravis Ultrasound; just going to be an asspain. If you want something better for your Windows games, a Monster Sound MX300 is revered for audio quality, but I've always had setup issues (I think I have a bad card). Old Audigy cards are cheap, too. You can have more than one in there.
RAM: RAM for old computers is cheaper than dirt. I bought a 256MB stick of PC100 last week for $4 off eBay. Just get two of those.
Storage: Noooooo SSD. Windows 98 has no idea what to do with it, and an OS needs to take different steps to manage it. Also, SSDs are all going to be SATA, and you need an IDE drive. Small IDE drives are still bouncing around, just try to get something that hasn't been used. Some people use a compact flash card, but I can't really see that lasting very long.
Drives: Optical drives are cheap. As long as it is IDE, you're good to go. I gut them from off-use Pentium 4-era business computers. Floppy drives...make sure it works? That's about it. 5 1/4" floppy might be kinda hard to find. Only thing I can really add is if you want to play DVD movies, you might need an add-in card.
If you buy smart, you'll hit waaaay under budget. Especially if you hit up e-waste recycling places. You might get lucky and get all the parts for a whole system for not much money.
Main: Ryzen 1700X / Gigabyte uATX board w/ PCI slot / XFX HD480 8GB / 16GB Corsair Dominator 3000 / Asus Essence ST
1999 Vectra: Pentium II 400Mhz / Integrated Matrox G100 4MB / Diablotek Radeon 7500 64MB PCI / 363MB PC100 / Aztech AZT2320 ISA