They seem to all leak eventually. If you look at the board every day, I think you'll catch it before there's much damage, but it would be easy to forget about it for a year or two, and then...
The ones I'm familiar with are Varta 3/V60R. Its a Ni Cad, so needs to be run every once in a while to hold a charge. They have little tabs on them, sometimes two on one end. Those tabs seem to be spot welded to the battery - I was unable to desolder them. I think you are supposed to desolder them at the board. I crudely cut mine out because it was leaking badly. The blue-green leaking crud looked bad, but didn't seem to actually do any damage once cleaned off.
You can still get new ones, but I'm not sure that they haven't been sitting for 15 years already. There should be a better replacement idea, but would probably require some soldering at minimum. Most newer computers use something like a CR2032, which is a different voltage (might be close enough) but the motherboard will try to charge it, which might make it blow up.
Best options I can think of so far are:
Cut the tabs off with some space for soldering
Solder some wires onto the board to run to a new battery
Run them to a different battery mounted somewhere in the case - either a real NiCAD (anything with 3.6 volts should do), or possibly a modern cell with some diodes in series to prevent charging.
I've only done steps one and two and wasn't successful in using another battery, but at least stopped the damaged to the board and didn't kill it in the process. Attached is what it looked like.. don't laugh at my soldering job, it was awful (and one wire fell off). Anyway the board looks crummy but cleaned up ok.