You are where I was about 14 years ago. My target capacities have matured and I happen to be looking to ditch my AWE64 Gold and Voodoo 2 SLI (Diamond Monster 12MBx2). Neither are bad choices for their purpose and I have no regrets picking them up: it is more a matter of what I was looking to cover while keeping everything in one case.
I happen to be tackling the hard drive alternative question myself right now as I know mine will not last forever. When it comes to peripherals, unless you are referring to game controllers (i.e. Microsoft Sidewinder) the Windows 98-era difference between USB & PS/2 is pretty much moot which is why I opted for the latter...besides, I like actually using those PS/2 ports that came free with my motherboard. If you wish to use USB peripherals there is nothing against that either: adapters are cheap online or in a vintage computer parts store, typically come in those pretty code colors, and sacrifice little to no capability (caveat: use a proper retro mouse so you know it speaks PS/2 as well as USB; not an issue for keyboards since they HAVE to work for the computer to function regardless of connection protocol).
System 20: PIII 600, LAPC-I, GUS PnP, S220, Voodoo3, SQ2500, R200, 3.0-Me
System 21: G2030 3.0, X-fi Fatal1ty, GTX 560, XP-Vista
Retro gaming (among other subjects): https://baronsfel001.wixsite.com/my-site