First post, by truth_deleted
Attempted to add a VNC server function to win32 dosbox by use of the libvncserver library. There are two paths, one is a vnc video output mode in SDL and the other is by patching dosbox only. Both developed by others.
To try either of the two methods, it was necessary to build libvncserver (attempted by mingw32/gcc4 and by cygwin). Cygwin has recently undergone internal changes which have affected the compiling and linking of libraries, the workarounds include extensive edits to source code of the libraries or reverting to an older version of gcc. Further, the libvncserver already built by cygports does not have posix threading. However, the mingw build of libvncserver was built successfully after some edits, both with and without the threading library. The vnc server examples seemed to work, but I was limited by my network setup and an inability to connect to a vnc server on localhost (confirmed this was not a libvncserver issue since stand-alone vnc had worked between computers on my local network).
Given the libvncserver was in fact a working copy, then the next step is method #1, using SDL (SDL_VIDEODRIVER=vnc). Tried this way a number of times until finding that mingw is not fully compatible with posix sockets. However, this method #1 should work in linux (and perhaps OSX). I can advise others if they'd like to attempt this. This method includes the ability for clients to connect with keyboard and mouse, but no sound is included in this vnc protocol. As an aside, it is possible to port the posix sockets functions by use of the Windows winsock2 library.
Method #2 involves dosbox and libvncserver only, while SDL is not specifically patched. This method was first developed at a time where x11vnc, not libvncserver, was the name of this vnc server library, so this patch is presumed to have been developed and tested in a x11 environment. Therefore, it is worthwhile to test this dosbox patch in linux (and OSX), but note that the VNC clients are capable of read-only access on the server. This method did not work for win32 dosbox, the window would blank where the vnc option was active. Since there is an interaction between SDL, libvncserver, and dosbox, my guess is that the threading was not fully functional and dosbox was unable to start the vnc server. There is some support for this since the libvncserver authors suggest building libvncserver without SDL and pthread support, a suggestion derived from their past experiences which also agrees with my attempts. In addition, there are posts on their "forum" about the difficulty in building libvncserver with mingw32, instead the authors suggest cross-compiling via linux or possibly using the cmake process to build via Visual Studio (I tried the cmake process with similar results as mingw32).
The disadvantages are the likelihood of lost vnc connections over the internet, lack of audio, and a library without extensive use by Windows users. Using the built-in network functions of games is much better, although there are games without network capability. In this case, at least for Windows only users, a VNC server and client are available as stand-alone programs. This should also allow for video hardware support of which the SDL and dosbox methods do not have. I don't recall reading any recent posts about use of such software, so I assume this is not an all-important feature for dosbox users specific to Windows.