I'm a bit rusty but I think that's possibly interfering with some linux defaults that are used to switch consoles. i.e. another login/shell

Map only the F5 key to Video, then select all the modifier keys (mod1, mod2, mod3) from the bottom left corner.

Ctrl + Alt + Fn changes the TTY on most Linux desktops, you might also want to change the modifier keys or use something like Ctrl + Shift + F5 (without Alt).

In Linux, the default for VIDEO CTRL-ALT-F5 opens a login, so it has to be remapped to work with DOSBox. I tried remapping to all kinds of combinations of keys. I can't get it to recognize anything to make the VIDEO work. Whatever I save, the line for VIDEO is empty but the screen output still shows it as CTRL-ALT-F5. I'm not understanding why it's so hard to use. Is there documentation of the file format? Maybe I could just enter it manually.

I think I figured out the key mapping file format. I did a random change so the mapping would be saved, then edited the file directly. For my system, "mod3" for SHIFT doesn't seem to work. Instead of the SHIFT-CTRL-ALT-F5 combination I saw recommended somewhere, I mapped VIDEO to CTRL-"Print Screen". Seems to work okay.

FYI, DOSBox can run my demo program from 1992 which uses direct screen access, changing of frequency of the clock timer, calling INT for mouse info, and setting CTRL-BREAK interrupt handler – good stuff! For anyone interested, the GitHub repo is here: https://github.com/Russell-S-Harper/EXPLORE, and a video of it in action is here: https://www.youtube.com/watch?v=XTOIfkqW9O0. Thanks everyone for your help!