Reply 600 of 652, by superfury
OK. Some more testing using testsuites and double checking using the Bochs source code as proper documentation (the 80386 manuals don't seem to fully specify the RETF (E)SP behaviour for example, nor how the immediate value if used affects it during and after the protected mode logic).
It appears that the increasing of (E)SP happens in all cases, with it also happening on the source stack of the higher privilege level procedure (to get the (E)SP and SS popped).
It did reveal a bug in the call gate system implemented in UniPCemu. When it was performing a RETF from a 16-bit procedure (using common 8086 segmentation logic, so 16-bit RETF (fallback) instructions that's using a 16-bit operand size) it would use various counters to handle the stack in the common 8086+ and 80386+ logic. But those two methods have different bases on their handling of stack pops (due to cycle-accurate handling differing between the two handlers). So it would work properly with 32-bit operand size (as it was using a correct base of 6), but on the 16-bit operand size (using the common 808x CPU logic for the RETF instruction) it would get a wrong base (it's past 6), thus causing the return stack pointer and segment to never be popped and simply assumed already popped into the local CPU buffers for handling. Thus whatever was in those buffers (SP reading buffer, ESP reading buffer and common 16/32-bit SS reading buffer) would be used (SP buffer containing zeroed data, ESP would probably be correct, but I'm not certain that's always the case,.SS would be the correct value for 32-bit RETFD, but SS would get the last 32-bit RETFD when a 16-bit operand size used. And SP for 16-bit operand size RETF would be zeroed or garbage (depending on initialized memory, usually zero (since the CPU structure is initialized with zero-fill))).
Edit: Hiren's boot CD doesn't seem to use inter-privilege RETF or call gates, but it does seem to use same-privilege level RETF in protected mode.
It still crashes unchanged.
Author of the UniPCemu emulator.
UniPCemu Git repository
UniPCemu for Android, Windows, PSP, Vita and Switch on itch.io