If (as seems likely) you're using a .conf file to change things, I'd suggest trying to run dosbox without it, just in case some change is causing problems.
There are probably other things others can suggest, but I've never needed as I haven't had too much problem with switching to a fullscreen, so I can't really say.
Personally, if you compiled dosbox from source and are still using the version you compiled in Fedora 11, I'd suggest removing it then recompiling, in case something that you compiled against in Fedora 11 has changed in Fedora 12.
If you installed dosbox via a rpm, or used yum, I don't know if an update would properly update the dosbox rpm, depending on the means of updating used. So if you didn't change things yourself, it is possible that you're still using a version of dosbox built for Fedora 11, not 12.
The best way to check would be to run 'rpm -q dosbox' on the command line and see if it says something along the line of dosbox-0.73-1.fc11.i586 as opposed to something like dosbox-0.73-2.fc12.i686
If you've still got the Fedora 11 version, I would definitely try removing it, and then installing the Fedora 12 version.
Though it is possible that this is a graphics problem. Are you having any problems switching the resolution of your computer? Can you switch to a low enough resolution?