Good question. Back in the day, I would install all kinds of software on my 98SE machine, and only when it stopped working would I reinstall Windows. Now, I'm installing only games and benchmarking software, so I like to keep things simple and clean. I don't like to install more than a couple of sound cards at a time.
Some of the installs leave behind mixer programs that still launch at startup even if the card has been removed from the system. I don't think they do any harm and are easy enough to uninstall, but it makes me wonder what else has been left behind.
Some ISA cards will "claim" an IRQ or DMA that you might need for a PCI card's legacy mode. Device manager will tell you that the IRQ/DMA is in use, but it's really not if you have disabled the ISA card in Device Manager. You can override with a manual config, but it might be confusing to some dealing with it for the first time.
Some Sound Blaster Live! installs messed up my system so badly that I will never install another one unless I have the install CD that it was bundled with. I bought a couple of NIB Live! cards for this reason only. If you pick the wrong the ISO from vogonsdrivers, the card will either not be recognized or run with glitches.
If you think you will be experimenting with a lot of hardware in your new hobby, I second the recommendation to use some kind of cloning software to make a "clean" image of your 98SE install. I use Norton Ghost 2003. If you don't have a lot of modern games on your system, a restore takes only 5-10 minutes.