VOGONS


First post, by NewRisingSun

User metadata
Rank Oldbie
Rank
Oldbie

I was having problems getting past a particular anti-debugging trap (Everlock) that uses self-modifying code. I noticed that in some situations, the prefetch in the 386_prefetch core did not seem to actually prefetch. Then I noticed that cpu/core_prefetch.cpp #includes core_normal/prefix_*.h, which have "cpudecoder=CPU_Core_Normal_Trap_Run;" hard-coded into them. As I understand it, this means that as soon as the trap flag is enabled, the emulation switches from the prefetch decoder to the normal decoder and then stays that way. I then changed all instances of "cpudecoder=CPU_Core_Normal_Trap_Run;" to "cpudecoder=CPU_TRAP_DECODER;" in those #included prefix_*.h files and defined CPU_TRAP_DECODER as "CPU_Core_Normal_Trap_Run" in core_normal.cpp, and as "CPU_Core_Prefetch_Trap_Run" in core_prefetch.cpp, and then the problem disappeared. Is this correct, then, or am I misunderstanding something here?

Last edited by NewRisingSun on 2016-09-13, 20:27. Edited 1 time in total.

Reply 1 of 4, by peterferrie

User metadata
Rank Oldbie
Rank
Oldbie

When the trap flag is set, an interrupt occurs after each instruction is executed, causing a queue flush each time. There is no longer any real prefetch queue active at that point.

Reply 2 of 4, by NewRisingSun

User metadata
Rank Oldbie
Rank
Oldbie

That's not the point. The point is that the emulation does not return to using the prefetch queue even after the trap flag has been cleared. This means that a debugging trap first using the trap flag, then after clearing it has some self-modifying code, will fail to executive properly.

Reply 3 of 4, by peterferrie

User metadata
Rank Oldbie
Rank
Oldbie

Ah, I overlooked the "and then stays that way".

Reply 4 of 4, by hail-to-the-ryzen

User metadata
Rank Member
Rank
Member

Is it possible to define CPU_TRAP_CHECK as 0 (core_prefetch.cpp) and obtain the same result?