Yeah, I made it working via a semi-random registry hack!
I have found that registry key
[HKEY_LOCAL_MACHINE\SYSTEM\ControlSet001\Control\Class\{4D36E968-E325-11CE-BFC1-08002BE10318}\0001\Settings]
holds settings of Matrox Monitor Wizard which is then applied to Windows setting to override video modes into registry key
[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Video\{716C682B-24CC-41ED-8640-740F2C3A84F0}\0000]
Note that the GUID and folowing integer number is dependent on your system on how many VGAs you had installed before.
Under this key I located entry:
"D00.0.M00.Graphic.1600.1024.32"=hex:01,00,3c,00,40,00,00,00,58,0f,02,00,20,00,\
a0,00,30,01,03,00,03,00,28,00,00,00,00,00
And I just tried to duplicate it under name
"D00.0.M00.Graphic.1600.1200.32"=hex:01,00,3c,00,40,00,00,00,58,0f,02,00,20,00,\
a0,00,30,01,03,00,03,00,28,00,00,00,00,00
And guess what, after a reboot I got the desired 1600x1200/32bpp mode working 😀 But I was unable to set any other color depth for this resolution. I wonder what this byte sequence means and have no idea why it works with values from different resolution.
I looked closer and found that this line comes from here (the beginning of the line + some additional byte):
"Graphic.1600.1200"=hex:3c,00,4b,00,d0,78,02,00,40,00,c0,00,30,01,01,00,03,00,\
2e,00,0c,00,00,00,41,00,51,00,8c,ad,02,00,40,00,c0,00,30,01,01,00,03,00,2e,\
00,0c,00,00,00,46,00,58,00,48,e2,02,00,40,00,c0,00,30,01,01,00,03,00,2e,00,\
0c,00,00,00,4b,00,5e,00,04,17,03,00,40,00,c0,00,30,01,01,00,03,00,2e,00,0c,\
00,00,00,55,00,6a,00,7c,80,03,00,40,00,c0,00,30,01,01,00,03,00,2e,00,0c,00,\
00,00
As told before, this entry is the same for G200 and Millennium II
So i started with a copy of this bytes in
"D00.0.M00.Graphic.1600.1200.32"=hex:01,00,3c,00,4b,00,00,00,d0,78,02,00,40,00,\
c0,00,30,01,01,00,03,00,2e,00,0c,00,00,00
And this doesn't work, it took me back to 16bpp mode only. So I tried to change every Byte that differs from 1600.1024.32 entry, every change need a reboot and I found that the 10th Byte value of 78h is the root of problem. When I replaced it by 0Fh or 2Ch from other working entries it was fine and I had ability to choose 16 and 32bpp modes.
To proof my work I cleaned the registry keys of all matrox VGAs, uninstalled the driver and repeated the process from the beginning.
I intsalled the Matrox driver 5.82.018 again.
The newly created key
[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Video\{D28F452D-7619-4FEE-BFCE-E4A308DD3CA0}\0000]
lacks the Matrox override entries "D00.0.M00.Graphic.x.y.z" and only relevant entry was
"Graphic.1600.1200"=hex:3c,00,4b,00,d0,78,02,00,40,00,c0,00,30,01,01,00,03,00,\
2e,00,0c,00,00,00,41,00,51,00,8c,ad,02,00,40,00,c0,00,30,01,01,00,03,00,2e,\
00,0c,00,00,00,46,00,58,00,48,e2,02,00,40,00,c0,00,30,01,01,00,03,00,2e,00,\
0c,00,00,00,4b,00,5e,00,04,17,03,00,40,00,c0,00,30,01,01,00,03,00,2e,00,0c,\
00,00,00,55,00,6a,00,7c,80,03,00,40,00,c0,00,30,01,01,00,03,00,2e,00,0c,00,\
00,00
So I patched the 6th byte from 78h to 2Ch:
"Graphic.1600.1200"=hex:3c,00,4b,00,d0,2c,02,00,40,00,c0,00,30,01,01,00,03,00,\
2e,00,0c,00,00,00,41,00,51,00,8c,ad,02,00,40,00,c0,00,30,01,01,00,03,00,2e,\
00,0c,00,00,00,46,00,58,00,48,e2,02,00,40,00,c0,00,30,01,01,00,03,00,2e,00,\
0c,00,00,00,4b,00,5e,00,04,17,03,00,40,00,c0,00,30,01,01,00,03,00,2e,00,0c,\
00,00,00,55,00,6a,00,7c,80,03,00,40,00,c0,00,30,01,01,00,03,00,2e,00,0c,00,\
00,00
Rebooted and the display control panel let me choose 1600x1200/16 or 32bpp. On a fullscreen rainbow test pattern it was clearly visible the difference between hicolor and truecolor. I would like to fix the same problem in WinNT4 but there's no registry branch
[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Video\] at all...
All this issue seems to me like a Matrox driver/INF bug where they accomodated the videomode binary values to modern VGAs like G200 and newer and forgot that Millennium needs a different 6th byte. I wish really know the interpretation if this data to understand what I really changed but I'm hapy it finally works.