you can just cut the leads with a side cutter and take it off the board... and you should do it ASAP, there is already a bunch of corrosion forming on the battery terminals/solder joints, and not just should you remove the battery but also neutralize what has already leaked with something acidic (like white vinegar) and clean it off properly afterwards.

just use an external battery instead if you have a header anyways.

edit: cause you were asking me in a pm, it's more useful if it's in the thread for others to see :p
you don't have to cut "underneath" the battery, the leads aren't attached all the way down, you can just snip against the battery below the welding spots, though the way the contacts are shaped can make this harder as they seem to be getting wider on this one, so cut where they are thin and if you can't reach both sides easily you can cut one side and then just wiggle the other back and forth until it breaks, or wiggle it loose and then cut underneath.

This is very helpful. Thank you!

Is this a fairly low risk procedure or is there a chance I might destroy the board doing this?

It's very low risk, as long as one side is easy to cut first. Cutting the leads on the battery and cleaning with vinegar (as said by imi), then neutralising the acid with IPA or even distilled water is zero risk.

It worked!

Thanks all!

After checking to make sure that the BC12 header doesn't try to charge (it shouldn't) you could get one of those 3xAA alkaline holders for 4.5V, or a CR2032 holder on a header (3 volts). I've found on some boards the CR2032 works fine, while on others it's only enough voltage to hold the CMOS settings and not enough for the clock to advance.

Part number BH3AA-211 was exactly what you needed but looks like Digi-Key doesn't have them anymore. BHF-3A3 will also work as a solderless option but it looks like the leads are pretty short.