The amount of video memory needed can be calculated by the resolution used. For a e.g. 640x480 screen with 256 colors you need exactly 300 kb. If double-buffering is used, you need 600 kb. So for DOS games, you're pretty much fine with even 1 MB. But you really want 2 or 4 mb when using Windows. 8 mb on the Matrox is wasted if you don't use it for 3d. For 3d, all textures must be stored in video ram, so more is better.

Hard to imagine that too much ram could be a problem. 1mb min, 4mb is probably more thanenough. I’d be more concerned about drivers that support your monitors resolution, 2d acceleration in the driver, and raw dos performance.

A 486 or even a 586 will struggle with Duke3d. It's playable but not what I'd call smooth.
As already mentioned the extra video memory wont really help you in dos, In windows it'll allow different resolutions and colour depth the following site has a table with the combinations
http://www.danielsevo.com/bocg/bocg_screenres.htm

One downside is it doesn't have the best compatibility with duke at higher resolutions (Not that you would want to on a 486)
https://gona.mactar.hu/DOS_TESTS/

but on the plus side it's good quality image and the fact you can play the few games that support the MSI API
3D Accelerated Games List (Proprietary APIs - No 3DFX/Direct3D)