I have a couple of 486 motherboards that require 6VDC when using an external battery (it's specified in the manual), and they definitely do not keep the time with 3 or less cells. I don't think it's unusual, especially considering those 4AA external battery packs for PCs were quite common back in the day.
Besides, I remember reading a thread in the Vintage Computer Forums that there usually is a diode connected to the external battery input so there you have a voltage drop, and the input voltage for the circuits powered by the battery is well within operating values (don't have the link to that thread at hand though).
In any case, it would be good to check that the board isn't trying to recharge the batteries through the header you're using. Can you boot up without the battery plugged in and measure the voltage between the pins of the ext. bat header you're using?