When executing "retf 10h" to an outer privilege level, how is esp affected before and after the ss:(e)sp is popped? I assume it's incremented with the immediate before popping ss:(e)sp from the stack, but is the resulting esp incremented with the immediate as well?
The 80386 programmer's reference manual doesn't say to increase (e)sp after popping it from the stack, but according to https://c9x.me/x86/html/file_module_x86_id_280.html it is popped from the stack? Which one is correct?
Edit: Dosbox SVN (https://sourceforge.net/p/dosbox/code-0/HEAD/ … src/cpu/cpu.cpp , function CPU_RET) seems to suggest that both happen: (E)SP is increased before popping SS:(E)SP, but after loading it, the new copy of (E)SP is increased again after the stack switch? So it happens on both the stack of the RETF instruction, as well as on the stack that's on the outer privilege level?
You only add the count to the return esp because the pre-return esp is thrown away. Look closer at the dosbox implementation, they add bytes to reg_esp and never use it.
Well, it should be used at one point: UniPCemu uses it before popping SS:ESP during a inter privilege return that needs to switch to an outer privilege level(RETF where the new CS.RPL>CPL). It first adds the immediate number to ESP, then uses two normal POPs to POP ESP and then SS from the stack. Finally, then normal universal function handler adds the immediate to the ESP that was popped for the new ESP for the calling function.
Yes, I'd forgotten that the caller stack pointer is pushed before the parameters when passing though a call gate. Anyway, the callee esp is still thrown away after the return so handle it however you want.
Just found another little bug: When returning either throuh IRET or RETF to a lower privilege level(CPL gets higher), any data segment registers are validated and cleared(descriptor invalidated by clearing the present bit using IRET, invalidated by clearing the access rights byte using RETF, according to the 80386 documentation on https://c9x.me/x86/html/file_module_x86_id_280.html and the 80386 programmer's reference manual) and clearing the segment selector when they don't match the same criteria.
This is now handled in unipcemu using:
1 if (RETF_segmentregister) //Are we to check the segment registers for validity during a RETF? 2 { 3 for (RETF_segmentregister = 0; RETF_segmentregister < NUMITEMS(RETF_checkSegmentRegisters); ++RETF_segmentregister) //Process all we need to check! 4 { 5 RETF_whatsegment = RETF_checkSegmentRegisters[RETF_segmentregister]; //What register to check? 6 word descriptor_index; 7 descriptor_index = getDescriptorIndex(*CPU[activeCPU].SEGMENT_REGISTERS[RETF_whatsegment]); //What descriptor index? 8 if (descriptor_index) //Valid index(Non-NULL)? 9 { 10 if ((word)(descriptor_index | 0x7) > ((*CPU[activeCPU].SEGMENT_REGISTERS[RETF_whatsegment] & 4) ? (CPU[activeCPU].SEG_DESCRIPTOR[CPU_SEGMENT_LDTR].limit_low | (SEGDESC_NONCALLGATE_LIMIT_HIGH(CPU[activeCPU].SEG_DESCRIPTOR[CPU_SEGMENT_LDTR]) << 16)) : CPU[activeCPU].registers->GDTR.limit)) //LDT/GDT limit exceeded? 11 { 12 invalidRETFsegment: 13 //Selector and Access rights are zeroed! 14 *CPU[activeCPU].SEGMENT_REGISTERS[RETF_whatsegment] = 0; //Zero the register! 15 if (isJMPorCALL == 3) //IRET? 16 { 17 CPU[activeCPU].SEG_DESCRIPTOR[RETF_whatsegment].AccessRights &= 0x7F; //Clear the valid flag only with IRET! 18 } 19 else //RETF? 20 { 21 CPU[activeCPU].SEG_DESCRIPTOR[RETF_whatsegment].AccessRights = 0; //Invalid! 22 } 23 continue; //Next register! 24 } 25 } 26 if (GENERALSEGMENT_P(CPU[activeCPU].SEG_DESCRIPTOR[RETF_whatsegment])) //Not present? 27 { 28 goto invalidRETFsegment; //Next register! 29 } 30 if (GENERALSEGMENT_S(CPU[activeCPU].SEG_DESCRIPTOR[RETF_whatsegment])) //Not data/readable code segment? 31 { 32 goto invalidRETFsegment; //Next register! 33 } 34 //We're either data or code! 35 if (EXECSEGMENT_ISEXEC(CPU[activeCPU].SEG_DESCRIPTOR[RETF_whatsegment])) //Code? 36 { 37 if (!EXECSEGMENT_C(CPU[activeCPU].SEG_DESCRIPTOR[RETF_whatsegment])) //Nonconforming? Invalid! 38 { 39 goto invalidRETFsegment; //Next register! 40 } 41 if (!EXECSEGMENT_R(CPU[activeCPU].SEG_DESCRIPTOR[RETF_whatsegment])) //Not readable? Invalid! 42 { 43 goto invalidRETFsegment; //Next register! 44 } 45 } 46 //We're either data or readable, conforming code! 47 if (GENERALSEGMENT_DPL(CPU[activeCPU].SEG_DESCRIPTOR[RETF_whatsegment])<MAX(getCPL(),getRPL(*CPU[activeCPU].SEGMENT_REGISTERS[RETF_whatsegment]))) //Not privileged enough to handle said segment descriptor? 48 { 49 goto invalidRETFsegment; //Next register! 50 } 51 } 52 }
RETF_segmentregister is set to 1(otherwise set to 0) when returning to a lower privilege level loading CS during a RETF and IRET.
Since this wasn't being handled at all, it would cause the lower privleged layer to have a privilege escalation to a higher privlege level than they actually should have :s