Some boards from that era indeed handle >8GB of memory, but it wasn't extremely common (2-4GB was still very normal, so 8+GB is on the high-end side). Performance wise, I use a Core 2 Quad Q9550 as my main computer, and have a dual-core E6550 as an additional machine; both are still quite competent. The 9550 has 8GB of DDR3, and a GeForce GTX 660 (before that, Radeon HD 4800s in CrossFire); DX10 works just fine in the one or two games I have that can use it, I don't have anything that uses DX11 that I'm aware of though. The E6550 has a Quadro FX 1700 (GF8600) and 4GB of RAM, and handles the web and multimedia content just fine, but I haven't tried much gaming (in synthetic benchmarks like 3DMark and Aquamark it scores similarly to a GeForce 7900 though).
I think if you want to build a gaming machine, depending on what the board, PSU, case, your budget etc will let you get away with, you have lots of options - even if you want to do DX10/11. Alternately, put a "simple" graphics card in it (I like GF8600 because they're cheap and have complete h.264/flash/etc acceleration; the modern GT 610 would also be good) and have a fantastic web machine as-is. I wouldn't consider 8-12GB of RAM or a quad-core to be necessary for gaming - it really depends on the games though; if you're primarily running 32-bit DX9 titles the extra ram beyond 6GB is a waste of money (because none of them can use more than 3GB), but some newer 64-bit titles will favor the extra cores and require 6GB of RAM at minimum; C2Q and 8GB would be a good place to be.
Graphics card wise, the sky is really the limit - if you're trying to keep it era accurate, the Radeon HD 4000 and GeForce 200 series are where you want to live, but Radeon HD 2000 and 3000 and GeForce 8/9 are also good choices, as are newer cards. What you pick will depend on budget, power supply abilities, motherboard support (can it do SLI? CrossFire? neither? do you care about multi-GPU?), case size (some of the higher-end cards can be huge), and what it needs to actually do - if you're wanting to run something demanding (let's say Skyrim) at high resolution, you will want a very powerful card, but if you just want to trod around on the web and run lighter games, even modern entry-level cards may be suitable.