Interesting VM86 entering issue

Emulation of old PCs, PC hardware, or PC peripherals.

Interesting VM86 entering issue

Postby vladstamate » 2019-11-22 @ 22:23

So I am trying to write a piece of code that enters VM86. This is done from protected mode with paging enabled. What I do is push the CS, IP and FLAGS (which have VM bit set) and then execute an IRET.

I think some emulators(*) (including mine) treat that whole IRET as an atomic unit, but I think that might not be the case with all of them.

What I see is while the IRET is popping and setting up CS it somehow has not popped EFLAGS, and therefore not realized that VM86 is now turned on and instead of using segmentation it looks for the CS descriptor number. Since VM86 is using segmentation CS should be treated as a segment not as a descriptor ID.

What would be the correct behavior in this case?

(*) I've noticed this behavior with both QEMU and Bochs
User avatar
vladstamate
Oldbie
 
Posts: 962
Joined: 2015-8-23 @ 01:43

Re: Interesting VM86 entering issue

Postby Stenzek » 2019-11-23 @ 01:18

Potentially-silly question, but what is the effective operand size of the IRET? If it's in a 16-bit segment you'd need the operand-size override. Without it, that could explain the loading-CS-as-a-descriptor-rather-than-an-offset behavior.

The Intel manuals suggest it's all one operation. As far as I can tell, Bochs should behave this way too.
Stenzek
Newbie
 
Posts: 62
Joined: 2017-12-08 @ 08:30

Re: Interesting VM86 entering issue

Postby BloodyCactus » 2019-11-23 @ 04:00

have you pushed correct segment registers ahead of the iret? are you in ring0 when you iret? if your not it wont work.
--/\-[ Stu : Bloody Cactus :: http://kråketær.com :: http://mega-tokyo.com ]-/\--
User avatar
BloodyCactus
Oldbie
 
Posts: 945
Joined: 2016-2-03 @ 13:34
Location: Lexington VA

Re: Interesting VM86 entering issue

Postby superfury » 2019-11-23 @ 16:36

@BloodyCactus: Besides those, you also need the SS and ESP values on the stack for the return to V86-mode.

In total, on the stack you must have pushed(CPL must be 0 or else #GP(0)):
EFLAGS <- ESP. Must have VM set.
CS <- ESP+4
IP <- ESP+8
ESP <- ESP+C
SS <- ESP+10
ES <- ESP+14
DS <- ESP+18
FS <- ESP+1C
GS <- ESP+20

Although I don't know what happens to the ESP and EIP value when popped... UniPCemu doesn't truncate them to 16-bits(all others are automatically due to being segment register loads). What happens on real machines? The Pentium Programmer's reference manual says it loads them normally as 32-bit values into 32-bit registers? http://datasheets.chipdb.org/Intel/x86/ ... 143004.PDF Is that correct? Or are they truncated to 16-bits and loaded truncated into 32-bit ESP/EIP?
superfury
l33t
 
Posts: 3372
Joined: 2014-3-08 @ 11:25
Location: Netherlands

Re: Interesting VM86 entering issue

Postby vladstamate » 2019-11-25 @ 23:24

Thank you all. It was indeed a programmer error: the stack was not set up properly (as superfury described it) so the EFLAGS was not popped as I thought it should have been.
User avatar
vladstamate
Oldbie
 
Posts: 962
Joined: 2015-8-23 @ 01:43

Re: Interesting VM86 entering issue

Postby superfury » 2019-11-26 @ 06:28

Whoops. Looking again, the ordering should be EIP(+0), CS(+4), EFLAGS(+8), ESP(+C),SS(+10),ES(+14),DS(+18),FS(+1C,GS(_20).
superfury
l33t
 
Posts: 3372
Joined: 2014-3-08 @ 11:25
Location: Netherlands


Return to PC Emulation

Who is online

Users browsing this forum: No registered users and 2 guests