The most important thing with GPUs for Windows 9x is avoiding GeForce 6 and 7 cards.
Drivers are one thing, missing obscure features like table fog and paletted textures are other thing but biggest issue of these cards is that they don't do proper 16-bit dithering.
GF6/7 does dithering only of opaque polygons - throw anything transparent and it will render without dithering. Makes 16bpp completely unsusable in many games. Some especially very old games that used alpha blending sparingly might look reasonably ok with only occasional banding. In this sense GF6/7 is better than GF8 but far from being ideal or even adequate for early D3D/OGL games.
This is not only relevant for very old 16bpp-only games but even newer games because a lots of these do just look better at 16-bit. It is in the way texture texels visually blend in 16-bit mode versus 32-bit and often games just seems to look very different, like have slightly different lighting or something. Some games also just run much better at 16-bit even on cards which actually always render at 32-bit like Radeons do On the other hand GeForce cards up to FX benefit greatly from using 16-bit and at times its like you can increase resolution a notch or two or increase AA just because you render at 16-bit. It is actually a very nice side-effect of what for me became not "ah goddammit game don't support 32-bit color!!!" but I actually often prefer 16-bit + dithering and deliberately choose it. I just like how it looks and especially at lower resolutions like 640x480 or 800x600 with tons of AA (preferably SSAA that also does the same thing as AF to textures making resulting "AF level" being much higher than 8x on Nvidia cards!) dithering makes games with low resolution filtered textures look less soap-like. Heck, in some sense I like to even disable 22bit filter on 3dfx cards to get this pseudo-detail-texture effect.
Anyways, where it comes to Radeons they do have nicer 16-bit. Performance don't really benefit from using it other than some games (e.g. GTA San Andreas) just use much more CPU time for 32-bit Z-buffer processing than 16-bit). AA and AF is like on GF6/7 - in fact Nvidia just copied ATi's tricks verbatim. Radeons also don't support table fog in Win9x but you do get support for it in Win2K/XP so it is at least possible to dual-boot XP and run games which need table for which can run on XP that way. AA and AF looks better on Radeons. AA because ATi tweaked points from which they sample near-edges samples that gives better smoother results comparable to higher AA. AF is just because 16x AF is available but also ATi AF has generally slightly higher quality.
GPUs I recommend:
- FX5900XT - still somewhat available and prices don't hurt your walled and can be OCd, especially if you edit voltages in the card's rom. Much faster than GF4Ti and especially when using AA and AF. Use driver 45.23 so still quite good when it comes to Nvidia drivers.
- 4Ti 4200 - if you can find one that doesn't cost fortune... best non-AGP8x version as if you get card for ability to install older drivers you might just as well go for the one which supports older drivers even if slightly slower due to less bus bandwidth.
- Radeon 9700/9800 - cards from ATi's golden era...
- Radeon x800 - cards from past ATi's golden era that most people just ignored because everyone jumped DX9c/SM3.0 bandwagon despite Radeons being faster and cheaper and SM3.0 being for few years unused and is some instances some effects looking better without SM3.0 (like shadows in FarCry being smoother and such). Obviously what interest us today is these cards are much better for Win9x and older games in general. Also there is as far as I know no reason to get R3xx card over R4xx card for Win9x. ATi's drivers 6.2 "just work". Might be wrong on that but generally consensus seems to be that ATi drivers follow trend newer==better. Not sure about AGP compatibility but it looks like your motherboard should support all AGP cards so you should be good to go.
BTW. I would also recommend getting at least one GeForce 2 card. Can be MX400 or something like that to make it cheap purchase. It is just to have something that can use very old drivers to act as a reference in case of "is it Nvidia thing or driver thing or just not-3dfx thing?". Also GF2 cards still had texel alignment adjustment which some games need. For similar reason I generally recommend getting both FX and Radeon R300/R400 cards at once - just to cover all fronts. And of course for all retro-win9x systems that don't already use 3dfx as main GPU I recommend throwing Voodoo2, preferably 12MB to be true reference how game was intended to be played.