Try one of the other machine types. It supports a few different VESA chips now:

machine = hercules | cga | tandy | cga | tandy | pcjr | ega | vgaonly | svga_s3 | svga_et3000 | svga_et4000 | svga_paradise | vesa_nolfb | vesa_oldvbe

Only svga_s3, vesa_nolfb, and vesa_oldvbe machine types have VESA support built in; the other SVGA machine types need appropriate TSR programs to have VESA support. The default setting of machine=svga_s3 should have the best overall VESA support, although the _nolfb and _oldvbe variants may be useful in some cases.

Similar to DOOM engine games, Build engine games like Duke3D use scaled sprites for objects (items, weapons, characters), not textured polys, so I'm not understanding your description of the issue. Perhaps some screenshots would be helpful, because I don't recall seeing any glitchy graphics when using VESA modes with the game.

Ok I attached a screen. The problem appears to go away with nolfb as machine type but not with any others, including s3.

Unless you want to run Duke Nukem 3D as is, you might want to look into
using eduke32
http://www.eduke32.com/
combined with hrp (high resolution pack)
http://hrp.duke4.net/
It makes the game look much, much better.

Ken Silverman made the NOLFB program as a workaround for Build engine games running on the Windows platform where a LFB doesn't function correctly. I think that a LFB is generally preferable to paged video memory, but perhaps you will be satisfied with machine=vesa_nolfb as a workaround.

I've had no success in reproducing the glitch in the screenshot; so it seems likely that there are other factors involved than the LFB, such as settings in DOSBox or in the game. I've tried the 1.3D registered and 1.4 and 1.5 "Atomic" versions of Duke3D configured for the VESA 2.0 640x480 video mode; with default settings in DOSBox 0.74. Perhaps others that have the GoG release can report if they encounter similar glitches.

I recall that the build engine has some issues with 2 internal pages or something like that.
Think h-a-l-9000 investigated that. (although that could have been the HUD flickering)

I've noticed that the HUD will sometimes flicker after returning from the menu (brought up by pressing ESC) when the game is configured for 640x400; but the setup program doesn't offer that resolution, so you have to manually edit DUKE3D.CFG to get it. 640x400 has the benefit of no distortion in the HUD because the pixels are exactly doubled from the 320x200 resoluton the HUD was designed around.

Well as I mentioned nolfb did seem to eliminate the problem although I didn't use it extensively, as the problem always seemed to happen immediately anyway.

As far as the gog version goes, I don't know how it would be different or not. I actually just created a default .74 cfg file and replaced the one already there to do my testing with. The dosbox version it came with was outdated so I changed that. I haven't changed anything from defaults really except for the machine type.

Just posted another screen highlighting same problem. Using svga_s3.

Do fixed cycles help?

Fixed cycles does seem to work. I used 200000 with svga_s3 and didn't see any glitches.

Try using one of DOSBox SVN builds and increase the VRAM size to more than 8MB. This should eliminate the problem.