I "sort of" solved these issues after struggling with it for ages and figured I'd post about it just in case it saves someone (maybe me in a year or two?) some time.

System specs:
Ensoniq Voodoo Banshee AGP
Wintac W6BXA 440BX
Pentium 3 450Mhz
128MB SDRAM
20GB Maxtor D740x
Windows 98SE
HP P1230 CRT (Mitsubishi DiamondPro)

There are actually two problems here. One is that the latest reference 3dfx drivers for the Banshee (3dfx Voodoo Banshee V1.04.00) wouldn't work properly in Direct3D. They'd crash a few seconds into any D3D game. I looked around a bit online and only found one other place mentioning this. They solved it by using a newer\different driver. So I installed one of the many third party Banshee drivers on falconfly.de. I'm not going to be too specific because in the end I don't think the driver version matters much. I had no D3D support in dxdiag at this point with a fresh install of 98SE, so I then installed DX7.0a. After this, 3d acceleration in D3D games was fixed. But... I couldn't select 800x600. I went to the desktop and sure enough, 800x600 was completely missing. I checked the driver .inf file, assuming it was some mistake by 3rd party driver tinkerers, but I didn't see anything missing.

After reinstalling different drivers several times, I eventually got 800x600 to show up, but only in 16 color mode... that's 16 color, not 16bit. Needless to say, this was getting infuriating, because there was no obvious reason for this and it only affected 800x600 (which 3dmark 99 complains about terribly, and I use that as part of my testing suite). All other resolutions worked all the way up to 32bit color, but selecting 800x600 dropped it immediately to 16 color. I also noticed that I would never get any refresh rate options. I tried manually switching the monitor to different drivers (from Plug and Play Monitor), but nothing fixed these issues.

So what fixed it?

...

Switching monitors.

I switched to an old Dell 19" flat panel (my other tester display), reinstalled the monitor driver (always using Plug and Play monitor), rebooted the computer and 800x600 was available. I switched the VGA cable back to the HP CRT and it was displaying 800x600 at 24bit color as expected.

I don't know exactly what the cause of this is, but it looks like some kind of issue with the Ensoniq Banshee reading the EDID information of this CRT. I have no clue as to why it would have issues with this, since the Dell LCD is most certainly newer than this CRT from 2005. This issue wasted a ton of my time, but I just couldn't let it go. If anyone has any input to clarify what's going on here, I'd love to know more. I hope this post helps anyone else struggling with missing resolutions in Windows 98SE though. I'm sure I've had issues like this before with other cards, but this time I had to solve it. Thankfully, in the end the card worked perfectly. 😀

For what its worth, I found the "X-Mojomotion V1.0" driver to be the easiest to install and use without it doing anything weird but this isn't what fixed the missing resolution!