Test suite and/or how to find bugs in x86 VM

Emulation of old PCs, PC hardware, or PC peripherals.

Test suite and/or how to find bugs in x86 VM

Postby earlz » 2017-11-30 @ 00:32

Hi, so I have much more specialized VM than most projects here. Basically, we're making an embedded VM for use to run untrusted code in a specialized environment. The end result of this is that we don't need a full PC emulator. No hardware to emulate (since the environment is incredibly restricted), and also no ring0, gdt, idt, etc. We can make very broad assumptions, like a flat address space, no segment register usage, no "kernel" code that uses TSS and other system features.. oh, and also it's always in 32-bit mode. No need to worry about real mode (though we still implement address/operand size prefixes of course). Basically the intent is that you can compile a normalish program in C or whatever, and run it in this VM and get some output

So, the cool thing about this is that there are a ton of complex things we get to skip out on. However, the bad thing about this is that we can't just pop in Windows 95 and ensure that the opcodes are correct by it not crashing... Right now we are almost up to par with 386 support, with our eventual goal being 686 (though potentially skipping out on FPU support).

However, while making a toolchain for this custom environment we ran into several bugs... Fixed the easy ones, and now sometime after ~200 opcodes are executed it goes into bad logic and jumps to an invalid address. The invalid address comes from a load from the stack that is set significantly earlier.... so it's not trivial to figure out why the invalid address was loaded..

So, we're looking to basically implement a test suite of sorts that validates that the opcodes are implemented correctly. We just now designed a test suite template system that makes this somewhat easy.. but testing every single instruction in the i386 ISA is no weekend project, even when excluding the ring 0 opcodes.

I found this forum while googling for some issues around the AF flag, and figured I would ask this question here since there is a lot of interesting discussion happening (and this is literally the only x86 emulator community I've found) . Is there a test suite that is useable for testing single instructions without needing to execute some stub OS that our VM would not support? And what is the best route for detecting bugs in the opcode implementations other than making sure to have some automated testing? In addition, ensuring operation is correct is paramount. Once this is deployed, it is incredibly difficult to fix bugs for various reasons.

VM source code: https://github.com/qtumproject/x86lib
Example test code: https://github.com/qtumproject/x86lib/b ... _tests.cpp
earlz
Newbie
 
Posts: 2
Joined: 2017-11-30 @ 00:18

Re: Test suite and/or how to find bugs in x86 VM

Postby superfury » 2017-11-30 @ 01:47

Just took a look at it, I immediately see a bug in 16-bit and 32-bit parity flag calculations: it's always 8-bit, even with 16-bit and 32-bit operations, based on the lower 8 bits.

https://github.com/qtumproject/x86lib/b ... elpers.cpp
superfury
l33t
 
Posts: 2291
Joined: 2014-3-08 @ 11:25
Location: Netherlands

Re: Test suite and/or how to find bugs in x86 VM

Postby danoon » 2017-11-30 @ 02:08

Yeah, there probably isn't a quick fix for this. But once you get coding on the tests you will find it doesn't take as long as you think it will. I basically looked http://x86.renejeschke.de when designing the tests for each op code. My project was similar to yours in that I only emulated what an app would need, I didn't emulate the stuff an OS would use.

https://sourceforge.net/p/boxedwine/cod ... /testCPU.c
danoon
Member
 
Posts: 126
Joined: 2011-1-04 @ 19:12

Re: Test suite and/or how to find bugs in x86 VM

Postby vladstamate » 2017-11-30 @ 03:15

Welcome to the forum! Generally for testing our tests are designed as BIOS-es. As in they run directly on HW (emulated) and expect to be loaded when the computer powers on. They do not depend on existing BIOS. So as long as you can load a piece of code in memory and run it, those tests should work:

For 386 we've been using this one (written by the author of IBMulator): https://github.com/barotto/test386.asm
For 186+ you can use fake86's 80186 test (which I cannot seem to find on the net for now but I can provide you with the sources). Superfury do you have a link for that one?
User avatar
vladstamate
Oldbie
 
Posts: 802
Joined: 2015-8-23 @ 01:43

Re: Test suite and/or how to find bugs in x86 VM

Postby superfury » 2017-11-30 @ 05:34

vladstamate wrote:Welcome to the forum! Generally for testing our tests are designed as BIOS-es. As in they run directly on HW (emulated) and expect to be loaded when the computer powers on. They do not depend on existing BIOS. So as long as you can load a piece of code in memory and run it, those tests should work:

For 386 we've been using this one (written by the author of IBMulator): https://github.com/barotto/test386.asm
For 186+ you can use fake86's 80186 test (which I cannot seem to find on the net for now but I can provide you with the sources). Superfury do you have a link for that one?


Sure: http://orbides.1gb.ru/80186_tests.zip (Taken from http://forum.osdev.org/viewtopic.php?f= ... 9&start=15 ).
superfury
l33t
 
Posts: 2291
Joined: 2014-3-08 @ 11:25
Location: Netherlands

Re: Test suite and/or how to find bugs in x86 VM

Postby earlz » 2017-11-30 @ 18:10

After posting this I ended up finding the test386.asm program and one of my team members is going to basically rip out the system stuff and real-mode switching so that we can run it on the x86 VM. It doesn't exercise each permutation of each opcode (like it doesn't differentiate from add rm32, r32 and add eax, imm32).. but the connecting code is simple while the actual opcode stuff is where I expect most bugs to lie in. We'll do some basic unit test style testing to make sure the connecting code is correct for each opcode.

Also regarding the PF bit, I'm going to have to look that up. My understanding is PF was calculated on the parity of the entire piece of data that is the result, not just the lowest byte. I know of quite a few resources, but if you have any that are especially helpful for understanding flags and the subtle details, it would be appreciated.
earlz
Newbie
 
Posts: 2
Joined: 2017-11-30 @ 00:18

Re: Test suite and/or how to find bugs in x86 VM

Postby vladstamate » 2017-11-30 @ 23:42

It is just the low byte, Superfury is right. This has been proven by running against the 80186 test suite and test386.asm which were ran against actual HW. I am not sure why Intel designed it that way but it is no more weird than other strange Intel x86 stuff.

https://en.wikipedia.org/wiki/Parity_flag
User avatar
vladstamate
Oldbie
 
Posts: 802
Joined: 2015-8-23 @ 01:43


Return to PC Emulation

Who is online

Users browsing this forum: No registered users and 1 guest