Buy an Nvidia card? heh. Sorry...

You could try forgetting about fullscreen mode and just adjusting DosBox.conf to the same resolution as your desktop. Effectively just running DosBox in a Window.

I had the same problem.
Only solution I found was to get a different version of the drivers.
(Personally if you've got a real old card, at least 9200 or lower, I'd suggest using the DRI drivers instead of the proprietary ATI driver. There are some open-source drivers for newer cards just beginning to be reliable, but I'm not sure how many distros include them yet. And depending on your distro installing them may not be that easy or safe a job.)

Thanks for your replies. I think I will forget about fullscreen, as DosFreak said. I do not dare to change drivers after all I experienced with Ati Gfx cards.

I'm using ATI cards on Linux nearly since the dawn of time (or at least the dawn of Linux ^^). These days, it's a great hardware, as you get updates every month (it's an official AMD/ATI release schedule) and they are on the verge of implementing the last few missing functions people are missing, like AIGLX and the like. Plus, they have begun helping open source authors writing open source drivers for the very latest hardware, so ATI hardware is actually better supported than NVidia (at last... for years, it was opposite).

Since AMD publishes updated drivers frequently, try to use the very latest ones. If you can, use a pre-made installer for your distribution, then it will be painless. Don't use manual installation unless you have no choice: Accidentally mixing components of various drivers will produce problems and/or crashes. Your description sounds like such a case.

I own both, low- and high-end ATI cards (9600 and X1950), and both work flawlessly unless I accidentally mix files from different drivers/versions/packages.

Hint: the command "ldd /path/to/executable" will show you what shared libraries are used. Make absolutely sure that exactly the right "libGL.so" is used. Only the one that was shipped with your currently loaded fglrx kernel module will work correctly.

But as I said, most distributions offer a painless way to do that, by using the regular package manager. Absolutely use that if possible. Google if you don't know if/how it works on your distro.

And if your card is a Radeon 9200 or below, follow dougdahls advice, use the open source driver and not the closed source one. Again, try to use the distribution's package manager to install it.

I own a 9600 (XT), too and I used the specific driver package for my distro (Debian) as well. As I explained, I am using surface output, so libGL should have nothing to do with my problem. My concolusion is that I used the latest drivers, installed them with my package manager but fglrx does not work flawlessly.

Ah, surface: Disable "DGA" in your X server config. For the past few years, I haven't seen any machine where it works reliably, I guess it is broken for some common configurations.

OTOH, you're probably better off using overlay or opengl anyways, since those offer hardware scaling.

No, I tried and performance seemed best using surface. What does DGA mean exactly?

edit: I did not find any entry about DGA in my xorg.conf. Where do I have to look?

It's an acronym for "Direct Graphics Access", it exists in several versions and seems to suffer from API version problems.

Baiscally, it allows an app to get exclusive direct access to the frame buffer in full screen modes, which can make some kinds of programs faster (DOSBox might indeed profit from it, but most modern apps don't). Many graphics card drivers don't support DGA in conjunction with other features anyways, so it is quite common to see it disabled like this:

1Section "Module"
2        SubSection "extmod"
3                Option      "omit xfree86-dga"
4        EndSubSection
5EndSection

(You probably already have a Section "Module", and you might even have a SubSection "extmod". Do not touch any existing lines, just add the one Option line inside that SubSection, or add the whole SubSection if neccessary)

To check if DGA is available, you can use the shell command "xdpyinfo". Watch out for a line that says "number of extensions" and see if xfree86-dga appears in that list.

To check if you have Xv acceleration (for output=overlay), use the shell command "xvinfo". It will produce a short error message if not, and a long dump if yes. (Some drivers don't allow DGA and Xv at the same time. Xv is the better choice, as it is used more often)

To check if your OpenGL acceleration is working correctly, use the command "glxinfo" and search for the line "direct rendering: No" or "...: Yes". "Yes" means full hardware acceleration.

Thanks for your detailed description. I changed my xorg.conf as you explained but it does not have any effect. I even rebooted my PC to be sure. Xdpyinfo | grep -i "dga" gives my a nice XFree86-DGA. I also tried using this way of setting capitals in my xorg.conf but it did not work neither. Maybe I need to use the aticonfig tool. Btw there is now glxinfo on my system, it is fglrxinfo but this should be allright because it prints the right things.

glxinfo belongs to the mesa-tools package, but fglrxinfo is fine as well. It is basically glxinfo with less output, just the DRI settings are printed.

You might want to check /var/log/Xorg.0.log, it contains much information about X startup, for example which config file it is actually using, and what extensions it loads.

Finally I managed to get rid of DGA. But it did not fix my problem. Instead dosbox and fglrx keep crashing when changing to fullscreen using surface output. I also tried to get overlay working by telling aticonfig to use opengl or Xv as overlaytype but it does not change anything, neither. Overlay is very slow, even Pinball Fantasies runs slowly.

Maybe up/downgrade the SDL package.

I suspect that the fglrx driver might be having trouble with mode changes such as the resolution changing.
Unfortunately if so, short of changing the drivers to a newer/older fglrx or a completely different one, I suspect the only solution would be to not change the resolution.

If you can live without 3d accleration, you could try using the vesa driver instead of fglrx.

You could also try installing the open 3D drivers. I believe they might be in the xserver-xorg-video-ati package for Debian, but I don't know if your chipset would be supported for your version. Or depending on when the package was made, how stable/fast/good the drivers are.
Don't know how hard it would be to do the changes, or even how to do it for your distro.
I think the open driver would be either "ati" or "radeon".

Overlay and opengl do not cause any crashes but they are slow as hell. I think it if I could get them working everything would be fine. Maybe somebody has an advice how to get overlay or opengl running at a descent speed?

Hmm my ati 1950xt with the latest drivers seems to run fine with opengl in linux, although the overlay for video stuff doesn't seem to allow scaling. Although the drivers also mention something about being able to select overlay for opengl or for video, dunno if selecing one or the other will mean the other will be slow. Thought that might be more an issue with dualhead stuff or whatever. My old geforce under linux was definetly easier to work with, but who knows now with AMD in control things might improve more.

You should always enable Xv (= video overlay acceleration, this is what is usually meant with 'overlay', including DOSBox), as it is used a lot in home entertainment. Some OpenGL features disable it, I think it was OpenGL overlay and/or pseudocolor visuals. You don't need them anyways, as they are only used by hideously expensive upper-class 3D software, e.g. Maya.

My x1950pro works great, across two 1600x1200 heads, although there is one drawback: It seems like it only supports two 12-bit overlay formats (4:2:0), and not the 16-bit one (4:2:2) that SDL uses. I recall from my 9600 box that it has 16-bit overlay modes, so it seems to be card-specific.

BTW, another possible cause for crashes are other drivers. binary video drivers of all cards are known to have problems with framebuffer drivers for the same card. Using a VESA framebuffer is recommended.

As I already said, on both of my systems, nothing crashes ever, unless I have driver file mismatches. I use my distribution's driver packages and never install by hand. Fast and painless.

For a 9600, the open-source ATI drivers are an alternative, they are really stable as well, but no 3D acceleration (there is an open source 3D driver, but it's still too experimental for my taste).

After the last aticonfig --initial and disabling DGA changing to fullscreen in surface mode finally works. I dont understand why it works but I highly appreciate the fact it does. So many thanks for your advice and help. Overlay is still slow, but I am happy anyway.