I've just downloaded the source code for dosbox-0.61 and compiled it under RedHat 9. No special configure options, but I did set the environment variables CFLAGS and CXXFLAGS to '-O2 -march=athlon'. (I have a dual-Athlon 2400+ MP system.)

When I start dosbox, it briefly flashes a window and then aborts with the following message:

CONFIG: Using default settings. Create a configfile to change them
X Error of failed request: BadMatch (invalid parameter attributes)
Major opcode of failed request: 150 (MIT-SHM)
Minor opcode of failed request: 3 (X_ShmPutImage)
Serial number of failed request: 49
Current serial number in output stream: 50

Since I couldn't get dosbox-0.61 to work, I tried downloading and compiling the source for dosbox-0.60. That works. So, this problem must have been introduced between 0.60 and 0.61.

Browsing the file changes between 0.60 and 0.61, I noticed that OpenGL has recently been added. I tried recompiling dosboi-0.61 with the "--disable-opengl" option, but it didn't make any difference; I got the same X error message.