You can write any resolution you want in the resolution box (up to 16K), dgVoodoo will downsample it as long as Scaling mode is is set to "Stretched, keep aspect ratio". That is effectively SSAA.

I have not seen a game in dgVoodoo where MSAA works and resolution forcing doesn't. SSAA would likely have the same issues with games as resolution forcing.

I would try to keep both resolution forcing and downscaling factors as exact multiples of the resolution you're forcing from/to and the resolution you're downscaling from/to, but I've had good results even when not bothering with that.

Thanks that kind of worked nicely.

Set the game to 1280x720, the resolution in dgVoodoo2 to 5K and my monitor is 1440p.
The 3D now renders nicely in 5K while the 2D elements are sharp and big.

You can also set any in game resolution, and set DGV resolution to a scale of 2x 4x etc. This also basically serves as SSAA.
Actually here is a trick i use for many old game - they was designed mostly for resolution of 640x480 and whenever you set higher res in game, it does not scale menues, ui etc, and they are all getting small or completely broken.

So my trick to keep them all the way they was supposed to look is this:

1) set in game resolution to 640x480 or any other res - game was designed for (some are 840x480 and few are 1028x768, but there are weird PS1-PS2 ports designed for odd PAL and NTSC resolutions)

2) set DGV resolution to 4x or more. Set MSAA to 4x or more.

Basically game will think it renders in 640x480, but in fact DGV will render it in 2x or higher resolution but with proper scale of UI.
Thats the only true way to play many games.
Some games with tiled textures may have some gaps between tiles in menues after such upscale, thats something Dege has to fix in future version by toying around with texture coordinates re-calculations.