I'm not sure whose builds are currently using the vsync patch (if the one you're referring to is the one I wrote). To my surprise I noticed it in one of ykhwong's builds recently, but it wasn't functional due to a problem in the integration I guess. Perhaps someone else's build has it working. I don't think 0.65 or CVS have it.
Anyway, the vsync setting can't fix the problem. Rather than change the host's timing, it just changes dosbox's internal (fake) vsync timing -- which you can sync to the host's timing if you like for smoothness.
wiki: I presume you're using Windows. Have you tried running other apps fullscreen? Perhaps you should try running another emulator fullscreen - preferably one that allows you to select the resolution - and see if that has problems as well. My guess is that your drivers are set to attempt an unsupported refresh rate for certain resolutions. If you reinstall your lcd and/or gfx drivers, you may find the problem fix itself if you're lucky.
Edit: For a quick fix (without solving the underlying problem), you can try editing your dosbox.conf so that the fullscreen resolution is equal to your desktop resolution. That might set the same graphics mode as your desktop and allow you to use it fullscreen.