I had this issue for a while, first with a Sapphire Radeon HD 3850 AGP, then with a HIS Radeon HD 4650 AGP (DDR3). Both exhibited the same behavior. It's possible that all Radeon HDs within that range have the issue.

The cards would work under DOS like other video cards, but it has issues, as I experienced some graphical glitches with a number of games (e.g. Nova 9, ARYA VAIV, EPIC, and slightly in Wolf3D). Some relatively modern games, however, work fine with the card.

Apparently, when I tested its VESA functionalities using VESATEST, for both cards (HD3850 and HD4650) there were two issues:

1. The 320x200x8 mode is absent from the video BIOS, but 320x200x16 and 320x200x32 are present and okay.
2. The 800x600x8 mode is glitched, but 800x600x16 and 800x600x32 are okay.

The rest of the modes work fine for all color depths. It's possible that the games that work fine with the card do not really rely on mode 13H (320x200x8), or can be configured to use a better resolution (like DN3D, Quake and Terminal Velocity). Given the possibility of the BIOS implementations being similar among the cards of those generations, those two Radeon HD cards may not be the only one having the problem.

I'm wondering what video cards would do when a program calls mode 13H, but the intended resolution (320x200x8) is absent from the video BIOS. Had any other video cards had this problem in the past? It is possible that the video BIOS might try to emulate the intended resolution from a foreign one, hence causing the artifacts (should not be called glitch if this were the case)... Also, it looks interesting to me that the 320x200x8 resolution would be absent when there were over 50 available resolutions implemented in the BIOS and that the 320x200x16 and 320x200x32 resolutions were also available.

And now I'm asking if it's possible to add or fix resolutions to the system, as I recall UniVBE does this when it attempts to boost a card's VBE capability from 1.x to 3.0, but that's only limited to a set of old video cards. Also, Scitech Display Doctor was even able to fine tune those resolutions.

EDIT: The BIOSes of those AGP Radeon HD series (including the ones that I have, Sapphire Radeon HD 3850 and HIS Radeon 4650 AGP) can be found in TechPowerup's VGA BIOS collection. Apparently there were two BIOS revisions for the 3850 (one with a AGP-specific 9515 PCI ID, and the other with a generic 9505 PCI ID that allows installation of normal drivers), and the problem is present on both revisions.