On a Mac laptop try to press the fn key, additionally.

Otherwise, I think, the keys will be handled DOSBox-like in fullscreen mode.

I don't have a Mac, so I am probably wrong, but ...

For an X11-based system (which I believe Mac OSX is), it is the window manager (WM) that is first notified of the event that a key is pressed. It then decides what to do with that event (the keypress). In most cases the keypress-event will be forwarded to the window that is focused (DOSBox). But if the WM has already decided that key F1 does something else, then DOSBox will never see the key-event.

So start by looking at the manual pages for your WM, perhaps also the man pages for an application called xmodmap to modify the key-mappings.

No offense MiniMax - but that couldn't be more wrong. OS X has X11 as an accessory, but is not X11 based. It is somewhat based on the model called Display Postscript created in Nextstep, it's predecessor operating system:

http://en.wikipedia.org/wiki/Display_PostScript

Of course by predecessor, I mean what OS X was before Steve Jobs and Next Inc. sold it to Apple and eventually remade it as OS X.

Now back to the original post, this is easy to do and is explained as follows (for Leopard): Go to System Preferences -> Keyboard & Mouse. Under the keyboard "tab" there is a checkbox marked, "Use all F1, F2, etc keys as standard function keys". Check it and then close the window. This will reverse the behavior of your top level function keys - so that by default they act as function keys and you have to use the Fn key to access their special functionality.

Hope this helps.

well then minimax was right.
as you have to configure your system (s windowmanager) to don't touch those keys