Here's the HTPC system I was recently running, but I'm currently rebuilding:
ABIT KV85
Athlon 64 3400 Venice, socket 754
1GB RAM
XFX 7900GS AGP
ATI HDTV Wonder
Soundblaster Live! 5.1 Value (SB0220)
Running WinXP Pro SP3
In that configuration, I was able to do 720 pretty well, but 1080 was quite bad. 720 ran better with CoreAVC, but 1080 was still very bad. Fortunately, the TV I have it connected to is only a 720p model so it doesn't matter too much anyway.
I'm currently in the process of building a new image for this system. The only thing I've changed is the sound card; I have upgraded to an Audigy 2 ZS SB0350. I am not yet sure if the better sound card will have any impact on video playback performance.
According to this WikiPedia entry, NVIDIA has implemented PureVideo on models since the GF6 series. If you use VLC Player, it should take advantage of your card's hardware acceleration for video playback. However, with my 7900GS I didn't notice much difference between MPC-HC and VLC. They both played 720 adequately and 1080 poorly.
I'm quite interested to see what others have to say about YouTube performance in this thread. For a long time, I used to run a Radeon 9600XT in this same system and YouTube videos played fine. Sometime around 2012, something changed and they started to run like crap. This is actually what prompted me to upgrade to the 7900GS. However, now it too suffers poor performance with YouTube video.
So what exactly has changed and what is the solution? I intend to install the latest versions of Java, Flash, and Shockwave for my image. Maybe rolling them back post-imaging to versions from ~2011 will improve performance? Maybe running good ol' Firefox v3.6.28 will help? I'm not sure, but I can give it a shot and report back here later.
Maybe the cheapest/best solution for fixing this is to do as Skyscraper suggested and buy a Radeon 4350 or 4550. You can find 'em cheap on eBay. They do indeed support Flash H264 hardware accelerated decoding. They also compare to your 6600GT, so you shouldn't notice any decrease in gaming performance. Ideally, the Radeon 4670 is the much better choice but they're expensive, consume more power, and occupy two slots at a minimum. Your CPU will also likely be a bottleneck, so there's not much performance to gain by using the 4670. Since my ABIT KV85 is a micro-ATX board, I can't afford to sacrifice any of my PCI slots. Unfortunately for me, the 4670 is simply out of the question.