Hi,

I've been using DoxBox for about a year or two now. Recently, I got around to installing it on a newer system (Vista x64), on which I had enabled NX by default. DosBox crashes in this configuration (with the default DosBox.conf) when you launch a program, because it switches to the dynamic core and then blows up when trying to execute the generated native code (due to it not being protected as executable).

From five minutes of poking around on Sourceforge, it appears that the problem is in ``dosbox/src/cpu/core_dyn_x86/cache.h'', where it appears that ``malloc'' is used to allocate memory that code is written to (and subsequently executed from).

This is bad because the process heap (used in the default ``malloc'' implementation on Win32) does not contain executable heap segments by default, and will not return executable memory blocks.

A naive solution would be to reprotect the pointer ``malloc'''d, but this is less than ideal, since if somebody else gets a pointer on the same page (if your allocation was <4k), you are stomping on their memory protection (which gets even worse if you try and reprotect it back when you free memory, assuming you had two pointers to the same page checked out and both needed to be executable, after which you would be ripping out the exec protection on one block depending on the ``free()'' order - not a fun bug to track down).

A better solution would be to replace ``malloc'' entirely for purposes of allocations of code blocks on Win32, instead using ``VirtualAlloc(...PAGE_EXECUTE_READWRITE...)''. Alternatively, you could implement your own allocator on top of ``VirtualAlloc'''d regions that are always PAGE_EXECUTE_READWRITE. This may be advantageous as ``VirtualAlloc'' will return chunks of committed memory with a granularity of 4K (and will consume address space with a granularity of 64K, if you are not committing part of a previously reserved region, e.g. passing NULL as the requested allocation address). If you aren't doing a large number of small allocations that all need to be executable, it's probably not worth implementing your own allocator on top of ``VirtualAlloc'', though.

Actually, there's a third option, although it's pretty much an ugly hack (though if you really do have a lot of tiny allocations and don't want to implement an allocator on top of VirtualAlloc due to overhead, it might be worth considering): Create a private heap (``HeapCreate''), then protect each allocation to executable when you get it back from ``HeapAlloc'', never restoring allocations to their original protection. This assumes that the underlying heap allocator will never play with the protections on heap segment pages after returning pointers to them to you (which is probably a stable assumption). Unfortunately, this approach keeps the heap metadata marked as executable, which is still less than ideal. It at least gets you out of the stomping-on-process-heap-protections problem, though.

There are two workarounds for this issue:

1) Add DosBox.exe to the NX exclusion list (or don't enable NX by default). I don't really like this; NX is a good thing, but it's the most performant solution without requiring a code change.
2) Force the standard core, which is obviously less than ideal due to the significant performance hit.

The crash is immediately obvious if you run a program from DoxBox that triggers the dynamic core; it will blow up on the first dynamic core instruction, as EIP is not executable.