I think root of the problem is trying to play on GTX 260 with 8-16x AA. This is causing your high memory usage. This GPU is good for 4x AA. For 16x AA you probably want GTX 580. In period correct reviews they tended to use 4x AA.

There are 4 versions of GTX 260, the best one is core 216 rev 2 (GT200B), those often come OCed with small difference to stock GTX 275.

I tested last night Quake 4 to see how GTX 285 can cope with high levels of AA. The pictures below tell the whole story. Without AA around 600-700 Mb of VRAM is utilized. With AA 2X there is practically no difference; at 4X AA there is a very small increase in VRAM usage; with AA 8X it jumps to 700-800 Mb of VRAM used up. There is a huge increase the moment I select 16X AA to 900-1000 Mb and sometimes even more; the moment GTX 285 starts to use system ram, even with the game installed on a SSD, it starts to stutter - very rare, but it does: the game hangs for a fraction of a second and then continues and so on. Because this would impacts the results and also because GTX 285 can not maintain a 60 FPS locked experience at these settings so we would get a video card limitation, I decided that the maximum AA level for all games is going to be 8X! Only games based on IDTECH 4 engine have the option for 16X AA so games like Doom 3 will be tested at only AA 8X.
Settings: 1280*1024, ultra quality, advanced settings all ON, AA 16X ,VSYNC off:

Did you test 1600x1200 with 8x AA? Or just 1280*1024? More pixels has priority over very high AA.

Just 1024*768 and 1280*1024. I will test at 1600*1200 when I put back my only 21 inch monitor but I do not think VRAM usage will go over 1024Mb even in Quake 4 or Company of heroes - another game that loves ram as it takes around 2.2gb in a skirmish game on a large map.
I tried GTX 260 and it will not fit on the AX8 because several capacitors are blocking the card from being fully inserted into the PCI-express slot. I also tried radeon 4890 from Sapphire, 9800 GTX+ from Galaxy, GTX 460 that I borrowed from a friend and they simply do not fit... I will try tonight Radeon 5870 from Powercolor to see if I can install the card.

Finally found 2 cards that fit! Gigabyte GTX 460 fits just fine but requires drivers from the 2XX series and Powercolor Radeon 5870 also fits and can run using catalyst 10.2, the last good XP driver from ATI.
Now GTX 460 should be performing lower than GTX 285 - which is actually bottlenecked by the Athlon 64 X2 4800 - while Radeon 5870 should perform better.
What should I use? GTX 460 which requires a newer driver with unknown or questionable compatibility and performs worse than previous card or the Radeon 5870 which perform better, has a driver that I know is good but requires a full Windows XP reinstall as I do not want any remains from nvidia drivers to influence the results?