Why all the "unsure"? You have the hardware. Open up the case and get part numbers/model names etc to get sure. That would make it a lot easier to answer your main question...
- memory: look at the model numbers of the chips. That plus number of chips on a DIMM tells you everything you need to know.
- CPU: if slot, you should be able to see the info with the all-important SL-code on the top of the slot cartridge. If socketed, remove the heatsink and it's on the PGA package around the CPU core in full view. At the very least, check whether it's a slot or socket model.
- 10GB HDD: should have a sticker with brand, model etc
- ZIP drive: could be IDE or ATAPI (yes, two different types that behave differently, ATAPI is much more common), see sticker
- "generic sound card" undoubtedly has some info on it, at the very least check which chip(s) it has.
- NOS power supply might also be a problem. In any event it has a brand & model number
- motherboard is probably OEM, but guess what, there's info to be found on those things too. Find a model number and google it.
Can't judge the case without pics. Rust is nothing that usually can't be solved with Coca Cola or similar acids.
As for the caps - if you want to get into hardware this old, things will get more and more difficult (and expensive) if you don't get some basic skills around replacing caps. It's not rocket science. You can get away with a cheap-ish soldering iron (good 2nd hand is preferable over new crap), some thin leaded solder and a bit of flux. That's all you need to desolder the old caps and put in new ones. Any local electronics store should stock replacement parts, just go for the same specs (voltage, capacitance, pin distance) as the original ones and specify "low ESR" if asked in any more detail. No local store? All the big online places stock this sort of stuff. What works best depends on where you live, here in NL I prefer German company Reichelt for medium-sized orders, but there's a small local place with low shipping costs that's better if you really only need a handful of bits.
A "new" motherboard might work, but first you need to know what your're replacing. If it's regular ATX with regular power and front panel connectors, pretty much anything will work. If not, your choices may be limted, and by the time you've bought a new case, PSU, motherboard and video card, you're basically talking about a different system anyway.
As for Windows 98/DOS, the big question is what games/programs do you want to run? A P3-700 is VERY fast for DOS, but can be slowed down (lower FSB, disable caches etc) run most mid-1990s DOS stuff; early 1990s and 198s games will generally run too fast. Conversely, a P3-700 is fully capable of running Win98 games from 1998-2000, but after that the CPU will be quite anaemic. If you want mid 1990s DOS and late 1990s Windows games, this system would be fine. If you want more you may have a challenge covering all bases with a single system.
For DOS, the sound card is critical. DOS doesn't really have a concept of 'drivers', games address the hardware directly and so the cards need hardware compatibility with the games. If not, software (TSRs) can be used to emulate supported hardware, but at best it slows the system down and at worst you get problems with free conventional memory and compatibility. Bottom line is that you want an ISA (*NOT* PCI, unless you have PC/PCI/SBLink, which is another complicating technology) sound card with full (hardware) Soundblaster compatibility. Beyond that it's a matter of which games/era you're looking at, and how far down the rabbit hole you want to go. Given you're just starting on this journey. I'd recommend making sure you have Soundblaster Pro 2.0 compatibility, and leaving it at that until you have that working fully and you understand its limitations and what other options are available. It's entirely possible that that "generic sound card" already gives you this - but we'd need more info to be able to tell you that for sure.
SATA for P3 is simple: no P3 motherboard has native SATA as the last P3 motherboards are a few years older than the first SATA controllers. You could use a SATA controller card with a P3 (indeed I do exactly that) but be aware that there are both PCI bus revision and OS limitations. The only safe bet for Win98SE on any P3 board and SATA are the Promise SATA150TX2Plus and SATA150TX4 cards. Installing OS on them requires having a driver disk, so isn't that simple, particularly if your 3.5" drive is broken. A SATA SSD is the fastest storage you can use with a P3, but getting it working isn't elementary (OS drivers, partitioning on a modern system for correct sector alignment etc) and given your level of knowledge and experience, I'd recommend against it. Just go with that 10GB HDD, or if you want solid state, get a CF-card adapter.