I know this is a joking remark, but are you sure your OAK VGA is even slower than the original IBM EGA? I consider (Super-)EGA or early VGA cards to be period correct for XT-class machines. Your OAK likely has the OTI037 chip, which is likely on the high end of XT graphics card, and might even be equipped with a 36MHz crystal to support 800x600.
On the other hand, even XT-class machines can profit from fast graphics cards. Two of my three ET4000AX ISA cards that work in an 8-bit slot, and they work at no extra wait-states in my XT. The same applies to an ISA mach32. My OAK and Realtek cards don't manage to do that. So if your XT was upgraded to a fast after-market graphics card, the OAK might indeed slow the system down.