First post, by kjliew
Looking into DOSBox implementation of PageFaultCore() in paging.cpp, sound like at the very 1st page fault, it pushes the state into queue and switch to full core to decode the instruction. After some time, the full core decoder will return and a check is performed to find instructions to restart after page fault. If the match is found, it pops the state and switches back to dyn_x86 core. All is good.
Now the problem. If the match is never found, it keeps decoding the instruction with full core and one or more page faults can happen later which push the queue beyond its limit and DOSBox will crash. If it never hits the limit of the queue, then it will stuck with full core for instruction decoding and DOSBox will slow to a crawl on demanding games which typically require dyn_x86 core for full enjoyment. So increasing PF_QUEUESIZE can avoid DOSBox crash, but the performance hit remains.
In theory, this can happen whenever paging is employed. But for typical DOS games which use single-threads DOS extender stub lacking virtual memory support, I can understand that the likelihood can be close to nil.
However, this is the major road block for getting Win9x to work under DOSBox. I would even like to suggest that getting WFW 3.11 with 32-bit Disk/File access with native display drivers (not std VGA) could also expose the problem. In fact, if we can imagine that if a DOS game uses a multi-threaded DOS extender with virtual memory support, then such a game will either fail to run in DOSBox or slow to a crawl due to inaccessibility of dyn_x86 core unintentionally.
Thanks to h-a-l-9000, his paging patch somehow partly resolves the problem and is undoubtedly the ultimate key to get stable working win9x with DOSBox. However, h-a-l-9000 paging patch does not seem to solve all the page fault issues. It seems to be able to identify which page faults have a matching instruction re-entry point for restarting and which ones don't (which I still don't quite understand how he accomplished such marvellous tricks.) But sometimes, it also misses some and pf_queue never fully flushed. The whole story of this is non-deterministic, so I have to enable some logging in DOSBox to monitor pf_queue. Whenever I boot to Win9x desktop without pf_queue flushed, then I will just restart again. Once in Win9x desktop with pf_queue flushed, I never have any problem going forward. MechWarrior 2 Titanium-series and Half-Life both running well with Gulikoza Glide pass-through patch. An optimizing build (-O2 -fomit-frame-pointer) is less likely to have un-flushed pf_queue compared to debug build (--disable-core-inline -O0 -gdwarf-2). Similarly, running with more powerful CPUs also is less likely to get un-flushed pf_queue. (Core 2 duo vs Core-i5)
I would plead DOSBox devs to re-look into existing page fault handling issue and find ways to solve it elegantly. This is a low hanging fruit for DOSBox to support Win9x. I fully understand the complexity of DOSBox to support Win9x guest, but it looks like DOSBox already has the necessary pieces to be the best platform for emulating Win9x guest for gaming purpose, once the page faulting handling can be resolved.
IMHO this is equally important for DOS games. While DOSBox may not crash without having pf_queue flushed, it will slow to a crawl as most games designed to use DOS extenders are very CPU demanding.