Hope the above guide is useful to other fans of Quake II. Here's some background on how I discovered this approach.
I love Quake II to death and have played it a zillion times, but I've always struggled to get the brightness right. Fresh installs of the game are way too dark on most systems, and the built-in brightness controls have nasty trade-offs discussed below. I also have not had much luck with driver-level gamma controls. (Drivers vary in the level of control they provide, and some old cards don't support gamma correction at all, so I don't see this as a viable solution.)
I've never understood why Quake II is so dark and murky to begin with. Quake III Arena doesn't have this problem! My guess is that Quake II was developed and tested on old video cards and drivers that rendered textures very differently. Maybe someone more familiar with Quake II's engine technology can shed more light on this (pun intended).
Anyway, I recently embarked on a fresh playthrough of the game and arrived once again at this scene:
Something is clearly wrong here. Look at the lava. See how washed out the texture is? There's no way it's supposed to look like that.
I tried messing with the three console variables I knew about: vid_gamma, gl_modulate, and intensity. The only way I could fix the lava was by setting intensity to 1.0:
Now the lava looks right, but the rest of the scene is waaaaay too dark. What about the other two variables? Here's what different values of vid_gamma look like with intensity set back to 1.0:
(left to right: 1.0, 0.8, 0.6, 0.4, 0.2)
UI elements start to look washed out at any value below 1.0, so this doesn't help us much.
What about gl_modulate? Here's what different values look like for that variable:
(left to right: 1.0, 2.0, 3.0, 4.0)
This seems like a good solution at first, but it turns out that high gl_modulate values severely compress the lightmap's dynamic range. There's a good write-up here that explains how the variable works and why it sucks. The loss of dynamic range is most apparent for objects (including the gun model) that are placed in dark areas:
So, another no-go. I propose that the optimal values for these three console variables are as follows:
So, how do we get the game to look right? As in, not too dark and not too bright? This is where Q2Gamma comes in. This is basically the Quake II equivalent of idGamma, a must-have gamma correction tool for GLQuake which is also way too dark out of the box (and also lacks in-game brightness controls of any kind).
I don't know exactly what Q2Gamma does. It doesn't touch the game's existing PAK files, so it must make the game apply brightness and contrast filters to the textures at runtime. Here's what the game looks like with Q2Gamma's built-in TNT preset:
Nice and colorful, but way too bright for me. Quake II is a gritty game: some areas are supposed to be dark. Q2Gamma makes the whole thing bright. Part of the problem is that it automatically sets gl_modulate to 4.0 (!). It also sets intensity to 1.0, though, which is telling. I think it's pretty clear at this point that the default intensity value of 2.0 is straight wrong; wonder how this got to be the default.
This is good, though. We now have a tool that can make Quake II brighter without blowing out the UI elements. What happens if we dial back the settings in Q2Gamma? I played around with this a bit, and finally arrived at the values listed in my first post.
Much better! I think this is about as close to the intended Quake II look as we are likely to get, at least without resorting to a source port. The lava looks right, the UI elements are nice and vibrant and not washed out, the lightmap has its full dynamic range, and the overall brightness level seems natural. The only drawback I've noticed is a loss of detail on some bright textures (e.g., Bitterman's arms and hands), but this is pretty minor and I think I can live with it. I'll definitely be using this fix from now on.