VOGONS


First post, by DosFreak

User metadata
Rank l33t++
Rank
l33t++

http://www.trnicely.net/misc/vista.html

xecutable images created for the DOS/Wintel environment (but not employing the Win32 API), using the GNU GCC compilers and langu […]
Show full quote

xecutable images created for the DOS/Wintel environment (but not employing the Win32 API), using the GNU GCC compilers and language standards, are subject to failure (or performance degradation) when executed in Microsoft Windows Vista, because Vista arbitrarily restricts the memory space for the GCC executable to 32 MB (33,554,432 bytes). Attempts to allocate more memory than this using the malloc(...) function (or related functions, such as calloc(...)) will fail. This limitation applies whether the application is executed with the "Run" command, within a Command Prompt box (DOS box), or with the Start command. This limitation does not appear in Windows XP, Windows 98, or previous versions of Windows and DOS; the exact same executable, running under Windows XP SP2 or Win98SE, is capable of allocating several hundred megabytes of physical memory (if present on the machine). The limitation appears to apply to any compiler and linker not employing Microsoft's proprietary Win32 API.

The operating environment exhibiting this incompatibility in Vista includes GNU GCC "C" 3.04, DJ Delorie's DJGPP 2.03, and Vista Home Basic 6.0.6000 running on a Dell E520 with a Pentium D 915 2.8GHz dual core processor, 1 GB of 533 MHz DDR memory, 800 MHz front side bus, and SATA drives. The limitation appears with or without the use of executable compression, and with or without the use of the DJGPP DPMI servers CWSDPMI (Charles W. Sandmann), CWSDPR0, CWSDSTUB, and PMODE/DJ. Since the problem does not appear when the same executable image is run under Windows XP or Win98SE, the problem clearly lies in Vista, and is not due to the vintage of DJGPP, GCC, their libraries, or their DPMI servers. The system is operated as a standalone, single-user machine, logged on as Administrator (Command Prompt run as Administrator), with no Internet connection. Nearly all security features are disabled, including User Account Controls, Windows Firewall, Windows Defender, and Security Center (except for a warning message).

A simple test code is provided in the file vista1.zip (46K), which includes the source code vista1.c and a DOS/Wintel executable vista1.exe compiled with GCC "C" 3.04, DJGPP 2.03, and CWSDSTUB.EXE. The test code attempts to allocate a large chunk of memory (40,000,000 bytes), using malloc(...); if it fails, it repeats the attempt, decreasing the amount by 1,000,000 bytes each time. Results are reported to the console. Run it from a Vista Command Prompt (DOS box), with the "Run" command, or with the "Start" command; the results will be the same---failure until the request is decreased to 33,000,000 bytes (or possibly less). However, if the same code is run in a Command Prompt session in Windows XP, Win98SE, or Win98, or in a standalone DOS session under Win98 or Win98SE, the allocation will be limited only by the available physical memory. The code can also be run with a command-line parameter, e.g., "vista1 100", which will start the allocation attempts at 100 million bytes.

I have spent several days looking for workarounds to the 32MB memory limitation, without success. I have investigated property sheets and system settings, combed through the Windows registry, searched through Microsoft's forums, searched through the GNU/Linux forums, and looked through numerous other forums with information on Vista. Not only does no one offer a fix, they don't even seem to be aware of the problem. Out of desperation, I even e-mailed Microsoft Support with the problem. As expected, I received a boilerplate reply offering to discuss the matter for a nominal fee ($120 per hour or more).

Thus far I have found no workaround for this problem. For many applications, it may not be a serious matter, but much of my work requires very large amounts of memory (arrays of several hundred megabytes in some cases), and I feel sure this is true for other users as well. Even if all your work is normally done on non-Windows platforms, it would still be unfortunate to lose as a distribution base all of the Windows-based systems on the planet (currently about 90 % of all systems, and if Microsoft gets its way, nearly all of those systems will eventually be forced to "upgrade" to Vista).

Essentially the same limitation applies to source compiled with the g77 Fortran compiler; one difference is that the code will usually fail immediately with the error message "Load error: no DPMI memory" if total memory requests exceed 32MB. I do not know if the problem appears in other environments (e.g., other editions of Vista or codes cross-compiled for DOS/Wintel with other versions of GCC). Proprietary compilers which inherently convert code to the Win32 API (such as Visual C++ or even Borland C++ 4.52 with PowerPak) do not exhibit this limitation; they convert calls to the standard "C" malloc(...) to calls to Win32 API functions such as GlobalAllocate(...) or VirtualAllocate(...), which are then linked in at run time from Microsoft system DLLs. I suspect the real problem may be that Vista is treating any application that does not call the proprietary Win32 API (either directly, or through compiler or linker translation) as a "16-bit" application, and is then applying the 32MB limitation for "security" reasons.

I would appreciate being informed of a fix for this problem, if one exists. However, I am not interested in wild guesses, vague suggestions, or discussions of Microsoft. My definition of a "fix" does not include installing and booting to a different operating system (e.g., GNU/Linux, XP, or Win98SE), migrating code to the Win32 API, or migrating from the DJGPP environment to the CygWin, MinGW, Microsoft Visual C++, Borland/Inprise, or Watcom development environments.

http://www.justlinux.com/forum/showthread.php?p=866761

No, it doesn't limit gcc programs to 32mb of memory. […]
Show full quote

No, it doesn't limit gcc programs to 32mb of memory.

What it does do (according to my psychic powers, given the information on that page -- i.e. this is just a guess, but I bet it's right), is restrict the maximum amount of memory available to any DPMI server (or perhaps an incorrectly-coded DPMI server?). The fact that the program works under 98 and XP (and likely 2000) does NOT mean the program is correct! It just means that 98 and XP either (a) contained a workaround for this known-broken program, which was removed in Vista, or (b) never used the extra memory, so they didn't care about allocating it using DPMI (this is generally the cause of issues with a program putting a non-zero value into a reserved parameter, for instance, but I don't think that's happening here), or (c) didn't care about DOS programs eating up all available physical memory. I'd guess the problem is (c) -- Vista probably started to care about old DOS programs eating up physical memory.

For various major known issues (at the time Win95 was released) with various DPMI-using programs, see bonus chapter 2 of Raymond Chen's book (PDF), entitled "How to Ensure That Your Program Does Not Run Under Windows 95". This issue is just more of the same.

The problem is DPMI ("DOS protected mode interface"), not gcc. The problem for this person is that they refuse to move out of the DOS world. So this person doesn't like win32; too bad. If you want to run on Windows, you'll have to use either the win32 API (which is documented) or the NT API (which is not). Just like if you want to run on Linux, you'll have to use either glibc or one of the other C libraries, or code syscalls directly (although on Linux, both are documented). There is no POSIX layer for Vista. (Well... there might be, but if it exists, it's only barely POSIX compliant, and it runs in parallel with win32, as a separate layer on top of the undocumented NT API.)

In short: DPMI is dead, and has been for about a decade now. And programs compiled with mingw or Cygwin gcc will work without this limit; it only exists if you use a (certain?) DOS extender.

I can confirm that this is likely the case since I have run into situations in NT4/2000/XP where various programs run in NTVDM would eat memory like there's no tommorow.

Now the question is what file controls this....I've already tried swapping with XP's DOSX.exe but that did nothing. Switched out NTVDM with XP's and 2003 NTVDM but that did nothing. (XP NTVDM didn't work at all, 2003's works but doesn't fix the issue).

I tried his utility out in XP and it does work with any memory value all the way up to 1+gb. In Vista it maxes out to 32mb....

Last edited by DosFreak on 2007-03-31, 21:18. Edited 2 times in total.

How To Ask Questions The Smart Way
Make your games work offline

Reply 1 of 19, by dvwjr

User metadata
Rank Member
Rank
Member

DosFreak,

If you get the chance can you send me the Vista (32 or 64) versions of DOSX.EXE? I do not run Vista, however I could take a look to see if the Vista DOSX for its NTVDM is where the the problem lies.

I'd like to disas, err 'look' at the Vista version of DOSX.EXE, just to be sure. 😁

Thanks again,
dvwjr

Reply 2 of 19, by DosFreak

User metadata
Rank l33t++
Rank
l33t++

You've got PM!

It looks like XP/2K3 use the same DOSX.EXE at least as far as size is concerned: 53, 840 (Haven't done a windiff yet)

Vista Dosx.exe is smaller. 53,536

How To Ask Questions The Smart Way
Make your games work offline

Reply 3 of 19, by dvwjr

User metadata
Rank Member
Rank
Member

DosFreak,

Thanks for the DOSX.EXE from Vista. Does not look like it limits the DPMI memory available for the NTVDM. The Vista DOSX.EXE works fine in the WinXP (SP2) NTVDM - it does not limit DPMI memory requests above 32MB as this individual encountered on Vista. Nothing quickly visible in the DOSX.EXE looks to limit DPMI requests in the BOP calls...

The only other thing might be the HIMEM.SYS from Vista, since the HIMEM.SYS driver must be loaded for DOSX.EXE to go resident. The WinXP (SP2) version is 4,768 bytes... If the limit is not in the Vista HIMEM.SYS, then Microsoft must have put a limit in the Vista kernel on any NTVDM memory requests. If you get a chance and have time I would be interested in looking at the Vista version of HIMEM.SYS.

In that rant so Nicely published, he also mentioned the following:

DOSBox 0.70 from SourceForge.net, installed under Vista, exhibits similar memory limitations.

It would be interesting to see if a 64MB DosBox v0.70 VM tested on Vista with his VISTA1.EXE DPMI test application requesting 65MB of DPMI memory would also fail. I do not have Vista so someone else would have to try this to see if even Dosbox v0.70 would fail or pass. I cannot see how DosBox v0.70 which does not request its memory from Windows as emulated DPMI would have the same problem he attibutes to the Vista NTVDM... 😒

Thanks again for your help,

dvwjr

Reply 4 of 19, by DosFreak

User metadata
Rank l33t++
Rank
l33t++

I sent him an email earlier today on DosBox and I tested his utility in DosBox. After changing the memory in dosbox.conf to 64mb the utility picked up that value just fine.

I told him if he wanted higher memory out of DosBox he'd have to modify the source.

Just looked at Himem.sys and they are both the same size between XP and Vista.

Looks like this is something else I'll add to my game compatibility list. Man, it's a good thing I stopped NTVDM compatibility testing....I'd go insane. Hopefully Vista will stop the NTVDM insanity but I'm sure someone will find a way to hack around this limitation.

Is it possible to force the extender to swap to the HD first before using real memory? That might be one way to do it. heh. Create a ramdisk. Have the extender swap to the ramdisk. 🤣. I can see it now. Someone will actually do it so they can play their DOS games in NTVDM! hahahaha. Man, I need to get out more.

How To Ask Questions The Smart Way
Make your games work offline

Reply 6 of 19, by dvwjr

User metadata
Rank Member
Rank
Member

DosFreak,

If you get a chance you please do a binary file compare on the WinXP (SP2) HIMEM.SYS vs the Vista HIMEM.SYS? The file sizes might be the same but there might be some byte differences nevertheless...

Thanks again,

dvwjr

Reply 8 of 19, by dvwjr

User metadata
Rank Member
Rank
Member

DosFreak,

Thanks for the news. Then it looks as if that Microsoft did limit the Vista NTVDM to 32MB of XMS and DPMI memory in the Vista kernel. Knowing Microsoft there is probably some undocumented Registry entry which could change the value.

Or maybe they really did hardcode that XMS/DPMI limit. 😒

From what I read about that "Nicely" guy, he was the one that first noted the original Pentium FDIV bug...

Thanks again,

dvwjr

Reply 9 of 19, by wd

User metadata
Rank DOSBox Author
Rank
DOSBox Author
DOSBox 0.70 from SourceForge.net, installed under Vista, is not subject to the 32MB memory limitation, apparently because it lin […]
Show full quote

DOSBox 0.70 from SourceForge.net, installed under Vista, is not
subject to the 32MB memory limitation, apparently because it links to the
Win32 API. It does have a self-imposed default memory limit of 16MB,
which can be increased to 64MB by editing the configuration file
dosbox.conf; to go beyond 64MB you would need to modify and recompile
the source. Thanks to Dos Freak and Daniel Verkamp for this information.

Reply 10 of 19, by DosFreak

User metadata
Rank l33t++
Rank
l33t++

Yeah, I don't know where he get's the WIN32 API stuff from. I guess he doesn't understand the concept of an emulator?

How To Ask Questions The Smart Way
Make your games work offline

Reply 11 of 19, by Kippesoep

User metadata
Rank Oldbie
Rank
Oldbie

No, it means that the limitation he found is apparently applied only to programs that don't link to the Win32 API. DOSBox does, as it is (compiled as) a Windows program. A console program that doesn't use the Win32 API (i.e. one that would also run in real DOS) would experience the 32MB limit (if I interpret the article correctly).

My site: Ramblings on mostly tech stuff.

Reply 12 of 19, by DosFreak

User metadata
Rank l33t++
Rank
l33t++

Now that SP1 is out has anyone tested this? Does anyone care? heh.

I do have SP1 installed so if anyone wants me to test it I can....not sure what the point would be though.

(The rumor was that this was to be fixed in SP1)

How To Ask Questions The Smart Way
Make your games work offline

Reply 13 of 19, by dvwjr

User metadata
Rank Member
Rank
Member
DosFreak wrote:

I do have SP1 installed so if anyone wants me to test it I can.

I care. 🤣

I don't have Vista, but if your Vista (beta SP1) can run the Nicely test attached to this post from his website, I would appreciate it muchly. 😁

The test default is for around 40MB of allocated DPMI memory, since the Vista RTM was limited to 32MB of DPMI memory. If the Vista (beta SP1) breaks the 32MB barrier, then it would be interesting to see what the new Vista (beta SP1) DPMI allocation limit is now by trying other memory amounts on the command-line. WinXP DPMI allocation limits appear to be limited to around the physical RAM minus most of the RAM occupied by the WinXP kernel.

Thanks for your time,

dvwjr

Attachments

  • Filename
    vista1.zip
    File size
    37.79 KiB
    Downloads
    897 downloads
    File license
    Fair use/fair dealing exception

Reply 14 of 19, by DosFreak

User metadata
Rank l33t++
Rank
l33t++

According to this forum SP1 didn't fix the isssue: http://www.bttr-software.de/forum/forum_entry.php?id=3630

Supposedly this utility overcomes the limit: http://www.bttr-software.de/forum/forum_entry.php?id=2870

Attachments

  • Filename
    hxntvdm.zip
    File size
    3.45 KiB
    Downloads
    909 downloads
    File license
    Fair use/fair dealing exception

How To Ask Questions The Smart Way
Make your games work offline

Reply 15 of 19, by dvwjr

User metadata
Rank Member
Rank
Member
DosFreak wrote:

According to this forum SP1 didn't fix the isssue:

Supposedly this utility overcomes the limit:

Looks like the hope for >32MB of DPMI memory under Vista (SP1) is kaput. I had found the same VDD for WinXP, the nice thing is that it does have the ability to limit DPMI memory requests, unlike the WinXP DPMI server. However, the above VDD utility will not help the problem unless the DPMI application is changed to use Int 31h/AX=0504h for memory allocation instead of Int 31h/AX=0501h.

Thanks again for the response,

dvwjr

Reply 16 of 19, by japheth

User metadata
Rank Newbie
Rank
Newbie

Hi,

I updated the tool mentioned above, and source code is now included as well.

I don't know if it helps with Vista, I don't own that "OS", but for XP it works. If someone is willing to do tests in Vista with it, I'll be glad to help.

http://www.japheth.de/Download/hxntvdm.zip

Reply 18 of 19, by japheth

User metadata
Rank Newbie
Rank
Newbie
Miki Maus wrote:

Can this program be used to fix Syndicate Wars eating up all free memory in Windows XP?

I don't know. If this is a DOS program, it's possible. In case it doesn't do what's expected - the source is included and it has full control about the amount of DPMI memory an application can allocate ...

Reply 19 of 19, by DosFreak

User metadata
Rank l33t++
Rank
l33t++

http://www.trnicely.net/misc/vista.html#Solutions

Registry Solution for Vista SP1 Rugxulo reports (3 June 2008) that the following workaround appears to eliminate the problem in […]
Show full quote

Registry Solution for Vista SP1
Rugxulo reports (3 June 2008) that the following workaround appears to eliminate the problem in Vista with Service Pack 1 (SP1). Use a registry editor (such as REGEDIT) to create the new (REG_DWORD) registry value DpmiLimit under the key

Computer\HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\WOW

Now edit the value to reflect the maximum amount of memory (in bytes) that you wish to make available to console applications (DOS command boxes). Thus, a value of 128000000 (decimal) would make 128 million bytes available to DOS boxes, as opposed to the default value of 32MB; a hex value of 0x7A12000 would have the same effect. It appears that 32MB is the minimum allowed; the maximum has not been determined, but is almost certainly 2GB or less. You will probably need administrator privileges to implement these steps, and you may have to reboot before the new setting takes effect.

This trick does not work in Vista releases prior to SP1 (such as my own version, 6.0.6000 from February, 2007); adding the new value to the registry has no effect, either before or after rebooting.
TSR/DLL Solution for all Vistas
Rugxulo also reported (29 May 2008) that a combination TSR/DLL available from Japheth at http://www.japheth.de/Download/HXNTVDM.zip, and targeted at the Windows XP environment, can in fact be used as a workaround for Vista's problem. Retrieve and unzip the zipfile. Put HXVDD.DLL into \WINDOWS\SYSTEM32. Put HXNTVDM.EXE somewhere in the path. When you wish to expand the amount of memory available to a DOS box (console or command box), run the TSR HXNTVDM.EXE within the command session. Again, you may need administrator privileges to carry out these operations, but you must not reboot. Applications which use DPMI calls to allocate memory will now have access to all of physical memory (but not beyond 2GB), not just 32MB.

This workaround should succeed in all versions of Vista. Note, however, that behavior in the DOS box may become erratic when the TSR is loaded and non-DPMI applications are executed.

How To Ask Questions The Smart Way
Make your games work offline