Reply 20 of 37, by fr500
I have made some progress in debugging this and was able to start WIN64 DOSBox with dynarec core into the DOSBox's DOS prompt. H […]
I have made some progress in debugging this and was able to start WIN64 DOSBox with dynarec core into the DOSBox's DOS prompt. However, launching a DOS program still ended up in
Summarizing the issues so far:
1. In gen_load_param_reg, the encoding of "mov R8, reg&7" and "mov R8, reg&7" are incorrect. From GDB disassembly, they are actually "mov reg, R8" and mov reg, R9".
2. For WIN64 ABI, space is allocated on the call stack as a shadow store for callees to save those registers. There are cases that gen_call_function_setup/raw were thrashing the return address in stack, and this was a major issue for
segmentation fault. One of the instruction is the "INT #imme8".
With those issues fixed, WIN64 DOSBox would boot up into the prompt on dynrec core. I believe there are more cases of stack thrashing for decoding other instructions. It is very tedious to track each of them down. Hopefully, this would give some insight to the DOSBox devs who are familiar with dynrec core to deal with issue #2 above once and for all.
Actually, not segmentation fault, it was a hung. That's what make it tough to debug. Otherwise, segmentation fault would have landed into GDB and stack trace could be helpful sometimes....
Hey, do you have a diff?
Have you made further progress?