VOGONS

Search

Common searches

Interesting VM86 entering issue

Topic actions

First post, by vladstamate

Posted on 
User metadata
Rank Oldbie
Rank
Oldbie

So I am trying to write a piece of code that enters VM86. This is done from protected mode with paging enabled. What I do is push the CS, IP and FLAGS (which have VM bit set) and then execute an IRET.

I think some emulators(*) (including mine) treat that whole IRET as an atomic unit, but I think that might not be the case with all of them.

What I see is while the IRET is popping and setting up CS it somehow has not popped EFLAGS, and therefore not realized that VM86 is now turned on and instead of using segmentation it looks for the CS descriptor number. Since VM86 is using segmentation CS should be treated as a segment not as a descriptor ID.

What would be the correct behavior in this case?

(*) I've noticed this behavior with both QEMU and Bochs

YouTube channel: https://www.youtube.com/channel/UC7HbC_nq8t1S9l7qGYL0mTA
Collection: http://www.digiloguemuseum.com/index.html
Emulator: https://sites.google.com/site/capex86/
Raytracer: https://sites.google.com/site/opaqueraytracer/

Reply 1 of 5, by Stenzek

Posted on 
User metadata
Rank Newbie
Rank
Newbie

Potentially-silly question, but what is the effective operand size of the IRET? If it's in a 16-bit segment you'd need the operand-size override. Without it, that could explain the loading-CS-as-a-descriptor-rather-than-an-offset behavior.

The Intel manuals suggest it's all one operation. As far as I can tell, Bochs should behave this way too.

Reply 2 of 5, by BloodyCactus

Posted on 
User metadata
Rank Oldbie
Rank
Oldbie

have you pushed correct segment registers ahead of the iret? are you in ring0 when you iret? if your not it wont work.

--/\-[ Stu : Bloody Cactus :: [ https://bloodycactus.com :: http://kråketær.com ]-/\--

Reply 3 of 5, by superfury

Posted on 
User metadata
Rank l33t++
Rank
l33t++

@BloodyCactus: Besides those, you also need the SS and ESP values on the stack for the return to V86-mode.

In total, on the stack you must have pushed(CPL must be 0 or else #GP(0)):
EFLAGS <- ESP. Must have VM set.
CS <- ESP+4
IP <- ESP+8
ESP <- ESP+C
SS <- ESP+10
ES <- ESP+14
DS <- ESP+18
FS <- ESP+1C
GS <- ESP+20

Although I don't know what happens to the ESP and EIP value when popped... UniPCemu doesn't truncate them to 16-bits(all others are automatically due to being segment register loads). What happens on real machines? The Pentium Programmer's reference manual says it loads them normally as 32-bit values into 32-bit registers? http://datasheets.chipdb.org/Intel/x86/Pentium/24143004.PDF Is that correct? Or are they truncated to 16-bits and loaded truncated into 32-bit ESP/EIP?

Author of the UniPCemu emulator.
UniPCemu Git repository
UniPCemu for Android, Windows, PSP, Vita and Switch on itch.io

Reply 4 of 5, by vladstamate

Posted on 
User metadata
Rank Oldbie
Rank
Oldbie

Thank you all. It was indeed a programmer error: the stack was not set up properly (as superfury described it) so the EFLAGS was not popped as I thought it should have been.

YouTube channel: https://www.youtube.com/channel/UC7HbC_nq8t1S9l7qGYL0mTA
Collection: http://www.digiloguemuseum.com/index.html
Emulator: https://sites.google.com/site/capex86/
Raytracer: https://sites.google.com/site/opaqueraytracer/

Reply 5 of 5, by superfury

Posted on 
User metadata
Rank l33t++
Rank
l33t++

Whoops. Looking again, the ordering should be EIP(+0), CS(+4), EFLAGS(+8), ESP(+C),SS(+10),ES(+14),DS(+18),FS(+1C,GS(_20).

Author of the UniPCemu emulator.
UniPCemu Git repository
UniPCemu for Android, Windows, PSP, Vita and Switch on itch.io

Go to top of page Go to top of page
Back to PC Emulation
Close window
Close window