Just found a few more little bugs in my emulation, reading the 80386 programmer's reference manual's instruction set chapters:
- LDTR faults during task switching were throwing #GP(TS) instead of #FS(TS)
- LEAVE was checking the top of the stack against segmentation+paging, while it should check the SS:(E)BP address instead. Thus any protection for said addresses would fail, as well as the paging unit possibly miscalculating the TLB entry for the access(perhaps even faulting when it wasn't supposed to, when a TLB miss and page fault occurs because of it).

Hmmm... Some of the stack instruction mentions some strange things. They either say #SS(0) when stack is out of bounds, before(all cases in UniPCemu), after(only with ENTER in UniPCemu. Also, why?) or before and after(also only with ENTER in UniPCemu. Also, why after?) the instruction? Anyone knows what that means?

Why would you check for invalid ESP after an instruction?

Enter:

#SS(0) if SP or ESP would exceed the stack limit at any point during instruction execution

Is that also when the instruction has pushed all it needs to push, before the next instruction?

Leave:

#SS(0) if BP does not point to a location within the limits of the current stack segment

Makes sense?

Pop:

#SS(0) if the current top of stack is not within the stack segment

What is "the current top of stack"?

Popa:

#SS(0) if the starting or ending stack address is not within the stack segment

Huh? Ending stack address? After the instruction completes? What about everything in between?

Popf:

#SS(0) if the top of stack is not within the stack segment

Again, what is "top of stack"? When is that checked(before, during, after?)?

Push:

#SS(0) if the new value of SP or ESP is outside the stack segment limit

Does that mean after the instruction finishes it's push?

Pusha:

#SS(0) if the starting or ending stack address is outside the stack segment limit

Again, see Popa.

Pushf:

#SS(0) if the new value of eSP is outside the stack segment boundaries

Is that another post-instruction check? After eSP is decremented and it's address written to?

Anyone?

It seems like it #SS(0) faults when some of those pushes reach the bottom of the stack for any possible push to the limit of the stack being reached because of an extra simulated push(without paging check) after those?

Sorry, can't help on/with this. 🙁
However, perhaps can other documents from AMD or Cyrix be helpful.
Or if whenever it's non-386 specific, check an older Intel document (say iAPX 286).
Modern Intel documents tend to confuse al lot of legacy stuff.

I have one little question, though.

With ENTER, the final (e)sp value is checked for validity(only segmentation, not paging), according to https://www.felixcloutier.com/x86/enter .
What kind of access is this? Is it a byte, word or dword access? Or is it the same as the operand size(16-bits or 32-bits)? UniPCemu currently checks against a byte memory access at (E)SP.

Edit: The iapx 286 manual seems to not mention the cases quoted in my previous post at all, except for the local (dynamic) variables of the enter instruction?