Ideally, desolder the whole thing. If you don't have the tools for that, snip off the leads for the battery.

As for how it will behave without a battery... I'm sort of assuming the battery isn't working great in its current condition, so: "pretty much the way it is now" would be a fair guess.

To replace, you can do a number of things:
- get a replacement Varta. It will work just the same and fail in ~20 years in the same way
- mod a CR2032 on there with a diode to avoid the board trying to charge it. Will fail sooner, but not leak like this.
- attach wires to where the battery was and use an external battery pack. 3x 1.2V AAA would work in basically the same way as this Varta. Mount it somewhere where it won't drip on vital electronics if it fails in future.

My experience is some machines that use a Dallas can detect when the battery is completely dead, and forcibly clear the CMOS on every boot.
I haven't run into this with barrel batteries. I suspect it's because the charging circuit masks the fact that the battery is missing, so the BIOS can't tell and doesn't care.
Even without a battery, the system may hold the settings for many minutes after shutting it off.
If you only use it occasionally, you could run without a battery, though you'll have to reset the BIOS settings every time you get it out to use it.

Thanks guys, I just needed some reassurance. I am happy to report that not only did it 'just work', it also auto detected a CF card on an adapter I put into it (that I had prepared in PCEM) and just booted straight into MSDOS 5. I am not ever touching that BIOS 🤣

If you haven't already you might want to pop out the BIOS chip and scrub the chip and area around the battery (maybe 3cm around it) with some white vinegar then rinse with water. Looks like some corrosion has started and battery fluid creeps everywhere. Take photos of the area then check it after a year or so to see if anything changes.