VM flag has precedence over the NT flag according to the Intel manual. I'm assuming real hardware functions the same, but you could always write a test to confirm.

Well, all documentation I can find says the VM flag has precedence over the NT flag. But that would make no sense, since you can use a task switch to switch to a VM task, but can't switch back when having done so if that's correct?

As far as I understand things, that's a limitation of the V86 mode. Plenty of other ways to exit V86 mode, I guess you could for example have a software interrupt which goes to a task gate.

What I mean is that you can't, for example, use V86 mode to handle an interrupt(e.g. MS-DOS) and then return(IRET) from it back to the caller when it's finished. You'll have to go the long way, installing an INT instruction to call the monitor at the IRET return address(e.g. MS-DOS style exit program that's trapped)?

I think what you're describing (handling software interrupts in DOS) is kind-of what the V86 Mode Extensions were added for. It allows you to specify a bitmap of interrupts which will not trap to the V86 monitor, instead branching to the location stored in the IVT (while remaining in V86 mode, even with an IOPL of 3).

See: https://github.com/stenzek/pce/blob/master/sr … u_x86.cpp#L2097 and https://github.com/stenzek/pce/blob/master/sr … u_x86.cpp#L2726. It also affects privileged instructions such as POPF/CLI/STI.

As far as I'm aware, Windows 95/98 do not use the V86 extensions, even if they are present. I remember reading somewhere that there was a reason (hardware bugs perhaps?). Not sure about NT-based OSes, I think they do. Pretty sure I used NT to verify my implementation of VME.

NT having lower priority than V86 is proven by EMM386 when loaded. If NT has higher priority, EMM386 crashes with a #NP fault(error #11) and asks for a reboot due to segment 11E trying to task switch incorrectly.

So, if the NT flag is essentially unsupported in V86 mode(only being cleared/set during task switches, but having no effect on IRET), is it affected by task switches to V86 tasks? So a CALL to a V86 task, will that set the NT flag of said task? Will it have effect on the V86 task(besides it being cleared when switching using an interrupt to another task)?

What happens when an interrupt occurs in V86 mode? Is NT always cleared in that case(except when switching to a different task because of it)?