VOGONS


First post, by NewRisingSun

User metadata
Rank Oldbie
Rank
Oldbie

I was having problems getting past a particular anti-debugging trap (Everlock) that uses self-modifying code. I noticed that in some situations, the prefetch in the 386_prefetch core did not seem to actually prefetch. Then I noticed that cpu/core_prefetch.cpp #includes core_normal/prefix_*.h, which have "cpudecoder=CPU_Core_Normal_Trap_Run;" hard-coded into them. As I understand it, this means that as soon as the trap flag is enabled, the emulation switches from the prefetch decoder to the normal decoder and then stays that way. I then changed all instances of "cpudecoder=CPU_Core_Normal_Trap_Run;" to "cpudecoder=CPU_TRAP_DECODER;" in those #included prefix_*.h files and defined CPU_TRAP_DECODER as "CPU_Core_Normal_Trap_Run" in core_normal.cpp, and as "CPU_Core_Prefetch_Trap_Run" in core_prefetch.cpp, and then the problem disappeared. Is this correct, then, or am I misunderstanding something here?

Last edited by NewRisingSun on 2016-09-13, 20:27. Edited 1 time in total.

Reply 2 of 4, by NewRisingSun

User metadata
Rank Oldbie
Rank
Oldbie

That's not the point. The point is that the emulation does not return to using the prefetch queue even after the trap flag has been cleared. This means that a debugging trap first using the trap flag, then after clearing it has some self-modifying code, will fail to executive properly.