VOGONS


First post, by superfury

User metadata
Rank l33t++
Rank
l33t++

My current version of x86EMU seems to overwrite it's own code (If I start Windows 3.0 inside of it, the VGA stops working because it's counter gets an impossible value instead of predefined constant). So it seems to be overwriting it's own code somewhere, but the Visual C++ debugger, nor the MinGW debugger can find it? How do I find the source of this invalid memory access?

https://bitbucket.org/superfury/x86emu.git

When I try to debug it using Visual Studio 2015 community, it tells me (after manually setting a breakpoint in the VGA) that the value (which is defined as a constant) set is an impossible high (32-bit) value. It also refuses to throw exceptions where the overwrite of the code happens, wherever this is I don't know and can't figure out.

Author of the UniPCemu emulator.
UniPCemu Git repository
UniPCemu for Android, Windows, PSP, Vita and Switch on itch.io

Reply 1 of 2, by vladstamate

User metadata
Rank Oldbie
Rank
Oldbie

It is unlikely that the executable overwrites itself. It is most likely a buffer overflow somewhere. You need to put a data breakpoint (either 8bit or 32bit) at the address of the constant. And then see as it will stop the debugger. I have same visual studio as you and I have done that operation and worked for me (as it was in my case I had an array of a size too small and I was writing to an index larger than the size).

If that fails (I don't think it will) you can use Valgrind in a Linux build of your emulator (or for windows check this http://stackoverflow.com/questions/413477/is- … ute-for-windows).

YouTube channel: https://www.youtube.com/channel/UC7HbC_nq8t1S9l7qGYL0mTA
Collection: http://www.digiloguemuseum.com/index.html
Emulator: https://sites.google.com/site/capex86/
Raytracer: https://sites.google.com/site/opaqueraytracer/

Reply 2 of 2, by superfury

User metadata
Rank l33t++
Rank
l33t++

Visual Studio won't let met create a Data breakpoint(it's grayed out). Maybe something about my Visual C++ 2015 project not being 'Native'? I can't find how to change my project to become Native...

Also my application uses Visual Leak Detector for detection, but it doesn't detect anything wrong at all as far as I can see.

Edit: Debugging it again with Visual C++ 2015 Community I notice that the GPU.emu_screenbuffer is pointing to deallocated memory?

Edit: After looking for the point deallocating the buffer, I found that the SDL helper function that deallocates the emulator's SDL surfaces was freeing the GPU.emu_screenbuffer (the main GPU buffer the VGA writes to). This was because when allocating the surface, the function that added the wrapper actually wrote the flags (which said that the pixels used in the surface shouldn't be deallocated using my memory allocation protection) to the SDL_Surface structure itself, instead of the wrapper(which is checked by the deallocation function). This caused the GPU render buffer (you can look at it as immediate rendered data from the VGA, ready to be resized for display) to be deallocated with the wrapper. So after one frame, the GPU render buffer did not exist anymore. This caused the VGA to check the memory (at the start of it's block rendering routine) to detect the GPU.emu_screenbuffer as bogus memory(memory which is deallocated), thus aborting all rendering(as it would corrupt memory and give an exception). This is now fixed and the VGA works as it should now again.

Author of the UniPCemu emulator.
UniPCemu Git repository
UniPCemu for Android, Windows, PSP, Vita and Switch on itch.io