Build your SDL library like this:
1cd SDL-1.2.12/
2make distclean
3./configure --prefix=/opt/dosbox --enable-static --enable-video-opengl \
4 --enable-video-cocoa --disable-video-x11 --enable-nasm --enable-cdrom
5make
(replace /opt/dosbox with the prefix you want to use)
And your DOSBox like this:
1cd dosbox-0.72/
2make distclean
3./configure --prefix=/opt/dosbox --enable-static --with-sdl-prefix=/opt/dosbox \
4 --enable-opengl --enable-fpu-x86 --enable-dynamic-x86
5make
.. this should enable OpenGL support.
Ah yes.. you should have set a few environment variables before:
1export PATH=/opt/dosbox/bin/:$PATH
2export CPPFLAGS='-I/opt/dosbox/include -I/usr/X11R6/include -I/usr/include'
3export CFLAGS='-I/opt/dosbox/include -I/usr/X11R6/include -I/usr/include'
4export CXXFLAGS='-I/opt/dosbox/include -I/usr/X11R6/include -I/usr/include'
5export LDFLAGS='-L/opt/dosbox/lib -L/usr/X11R6/lib -L/usr/lib'
.. as you could see you should also have Apples X11 (with its SDK) installed. Thats the place where the needed GLX headers are.