Try setting fullresolution=0x0 in dosbox.conf. Then it should not switch video modes when going to full-screen (i.e. it uses the same video mode as desktop).

It worked, but the screen size is too small, how can I fix it? Tried normal3x but with no success.

Change output= to a different value.

output=surface adds black borders. All the others (I use ddraw) zoom the picture to fit the screen (use openglnb if you don't like it blurry).

the other output options make dosbox reaaaaaally slow here. why is that?

"the other output options make dosbox reaaaaaally slow here. why is that?"

They use hardware acceleration. If Dosbox is slow with them, there is some sort of incompatibility between SDL (Dosbox's graphics library), your operating system, your video card, and/or your drivers. Did you try output=opengl? That worked for a friend of mine who has Vista. I have XP and output=ddraw works best for me.

edit: If none of the others besides output=surface work for you without slowdown, you'll have to go back to fullresolution=original if you don't want black borders. In that case you'll have to figure out why Dosbox or Windows sets a video mode that is unsupported by your display (out of sync means that either h-sync or v-sync is over the maximum or under the minimum). Windows XP has a checkbox called "Hide modes that this monitor cannot display" - Vista probably has something similar. Supported video modes are determined either by a monitor driver ("monitor INF") or by querying the display via DCC (signaling protocol used between the monitor and the video controller). My gut feeling is that DCC is used by default (my old Radeon had options to ignore DCC in case the monitor reports wrong values, which is sometimes the case).

they all make dosbox slow, really slow here, except for "surface". In case it helps, here's my current setup:
C2D E6400 @2.13Ghz
2.5Gb RAM
Motherboard Asus P5NSLI - nForce

Try a combination of fullresolution=original, output=surface, aspect=true, and scaler=normal3x.

If that still gives you "out of sync", try creating a custom resolution of 960x720, 85Hz (I have no idea how it works with Radeon drivers) and set your desktop to that before trying Dosbox in fullscreen.

I was thinking, does this "out of sync" error I'm getting has anything to do with ati driver?

BTW, i've already tried that combination you suggested, with no success.

"I was thinking, does this "out of sync" error I'm getting has anything to do with ati driver?"

Very likely. Dosbox works fine for a friend of mine who has Vista 64-bit and a Radeon HD video card so it's probably just a problem with settings. "Out of sync" means that your video card drives your display at too high or too low a refresh rate (quick googling reveals that your monitor supports up to 160 Hz vertical, 71 KHz horizontal - horizontal is basically just refresh rate times vertical resolution, e.g. 1024x768, 85Hz is slightly more than 65 KHz horizontal). You'll have to find where to limit the refresh rates (in the catalyst control center). Did you try the custom resolution trick?

edit: Try googling for catalyst control center refresh rates.
For example http://www.tweakguides.com/ATICAT_4.html has some information on how to set refresh rates.

edit2: I'm running out of ideas :^)

Ripa, I haven't tried that yet. What I can tell you, though, is that fullscreen used to work just fine on dosbox's original settings using nVidia on this very same setup (1024x768@85Hz). And I don't have Catalyst installed here, only the pure driver, as last time I tried it, it made my pc also very very very slow.

Ok, I updated the ATI driver, installed Catalyst and lastly, reinstalled Dosbox, and I'm still getting this "Out of Sync" stuff... I noticed that the problem is with horizontal frequency, which is @ 26.8Khz, the minimum allowed is 30 according to my monitor settings. Don't know what else to do, can anyone please help me? Tankan

Here's another shot in the dark: try to find an option called "reduced blanking" in the catalyst control center and turn it off. It's meant for flat panels. By the way, what is the vertical refresh rate when your monitor goes out of sync?

Hallo ripa. I did not find this "reduced blanking" option in catalyst. About the vertical frequency rate, is @ 60.0Khz, and the minimum allowed is 50. Tankan

Use "scaler=normal3x forced". Forced means that the scaler applies to text modes too. Of course if you play high-resolution games (that use something better than 320x240) then you need to turn off the scaler (or use normal2x), otherwise it will give you a too high resolution.

A better fix would be for you to force a higher refresh rate than 60 Hz (70 Hz would be the minimum in this case). You could try RefreshForce if you can't find anything in the catalyst control center.

I'm puzzled though why your drivers don't automatically use a higher refresh rate at low resolutions (in your case, it's trying to use 400 lines at 60 Hz, which is too low - the "standard" is 400 lines at 70 Hz). I haven't had a Radeon card in years, but from what I remember there was an option to get monitor EDID information straight from the monitor using DCC and then not allow video modes that don't comply with the monitor.

PS. the reason it worked with your NVidia card is that NVidia's drivers seem to add empty scanlines to a 400-line, 60 Hz mode to bring the horizontal frequency up to 31 KHz.

edit: If you don't care about text modes, you can set:
fullscreen=false
scaler=none
aspect=true

When your game starts (switches to graphics mode instead of text mode), you can go to full screen (alt+enter) and it shouldn't give you "out of sync". Only text modes give you "out of sync" because aspect=true doesn't affect them and so when you go to fullscreen when in text mode, it sets 720x400 60 Hz, which is too low for your monitor. Graphics modes are "corrected" by aspect=true from 400 lines to 480 lines, which brings the horizontal frequency up to 31 KHz.

There's some sort of refresh rate detection in the SDL code, that relies on
the respective backend (directx, windib) to provide the values for a mode.

Still no go...

I give up.