I bought a new in box Everex EGA card earlier this year and it had a bad tantalum cap on it that was shorted out and would prevent my IBM 5150 from powering on when the card was installed. Not surprising, considering the card was 25+ years old. The cap didn't pop or shoot sparks, it simply didn't work.
Just to get it done fast I replaced it with an electrolytic of the same value that I had lying around and it has worked flawlessly ever since.
As for other types of caps... Plan on replacing ones from the early to mid 2000s at some point, try to keep the system cool, clean and cleanly powered. The worst caps I've seen have been in systems that were clogged with dust and being powered by garbage PSUs that were also clogged with dust.
Other caps that I've seen go bad frequently are mid-90s SMD aluminum caps like the ones find on early Voodoo Graphics cards... IBM Model M2 keyboards seem to have a similar problem with these. I have a pristine looking M2 that is nonfunctional due to several traces on the flexible plastic key-PCB (can't remember the word for it) being blackened, likely caused by the leaking SMD cap I had to replace. I'd love to get it working because it is spotless externally... but that cap ruined the internals.