VOGONS


Reply 40 of 58, by mr.cat

User metadata
Rank Member
Rank
Member

Yes, comparing to the previous log (by exception counts) it seems a whole bunch of #UDs just vanished now.
The #PF counts also went down and actually there was a small difference in #SS and #GP too (one each gotten rid of).
I'm too uninformed to really make any judgment about the validity of these results though.
You may be onto something about finding instructions vs. actually executing instructions. The white paper mentions that the sifting takes place in ring3, so some instructions cannot be executed.

superfury wrote on 2021-07-04, 09:55:

... Anyone here got a Pentium II and is willing to run Baresifter on it ...

Just think of it as a yearly checkup, to put some stress on that old clunker to find out if it can still do the laps!
I'm doing this with qemu but it just isn't the same 😐

Reply 41 of 58, by superfury

User metadata
Rank l33t++
Rank
l33t++
mr.cat wrote on 2021-07-04, 22:35:
Yes, comparing to the previous log (by exception counts) it seems a whole bunch of #UDs just vanished now. The #PF counts also w […]
Show full quote

Yes, comparing to the previous log (by exception counts) it seems a whole bunch of #UDs just vanished now.
The #PF counts also went down and actually there was a small difference in #SS and #GP too (one each gotten rid of).
I'm too uninformed to really make any judgment about the validity of these results though.
You may be onto something about finding instructions vs. actually executing instructions. The white paper mentions that the sifting takes place in ring3, so some instructions cannot be executed.

superfury wrote on 2021-07-04, 09:55:

... Anyone here got a Pentium II and is willing to run Baresifter on it ...

Just think of it as a yearly checkup, to put some stress on that old clunker to find out if it can still do the laps!
I'm doing this with qemu but it just isn't the same 😐

What I meant is that the Baresifter testsuite only verifies instruction length itself (which seems OK) and skips undefined opcodes (not checking them at all) (stuff like undocumented 0FFF UD0 with/without modr/m not being handled at all since throws an #UD).
And besides that, it doesn't verify that the instruction actually performs the correct memory address for it's parameters (although that should be correct if the modr/m is parsed correctly). It also doesn't differentiate between memory and register operations in instructions (test386.asm does most of those. but still doesn't test all cases(for example ADD (opcode 00h) modr/m pointing to memory isn't tested at all, but it's registers version is(e.g. instruction 00C0))). Of course this also applies to most protected-mode semantics (stuff like different privilege level stacks, it's faults at different point during the privilege level switch (other than complete success or complete failure (so. e.g. a stack fault during those or page fault during those isn't tested when it happens halfway through)). And V86 mode tests are completely missing (and for those more modern OSes(Windows 2000 and up for example) stuff like VME is also required (which of course isn't tested at all as well)).

I don't know of any test suites that test those, except the test programs (some 5 programs) from Robert R. Collins which test those modes (VME in particular). It is apparently required to boot newer Windows versions (at least XP, since it crashes on VME on (early) Ryzen processors, unless it's been patched by now).
Edit: It's source code at least is found at http://www.rcollins.org/articles/vme1/
The executable link is still broken.

Author of the UniPCemu emulator.
UniPCemu Git repository
UniPCemu for Android, Windows, PSP, Vita and Switch on itch.io

Reply 42 of 58, by mr.cat

User metadata
Rank Member
Rank
Member

Right, right. Yes that would be the "more extended" part of the thread title, thx for spelling it out in a bit more detail.

Btw the exe package is there but the name is in lower case, it's just the link that's broken:
http://www.rcollins.org/ftp/dloads/vme1.zip

EDIT: Since I've been playing around with Bochs BIOS a bit, here's a little follow-up on that SeaBIOS commentary on the previous page...
The "legacy" variant does try to boot with i440FX+PureVGA, but I had to cheat a bit. It hangs at POST20 unless that part is commented out.
Note that VGAROM.BIN must also be one of Bochs' variants.

EDIT2: Yay, got the CD-ROM boot going by manually overriding the "bootfirst" variable (see rombios.c). Looks like plop is not necessary with this kind of setup.
Don't wanna jinx it, but I'm sure the trials and tribulations don't stop there...

Attachments

  • Bochs_BIOS_in_UniPCemu.jpg
    Filename
    Bochs_BIOS_in_UniPCemu.jpg
    File size
    83.75 KiB
    Views
    1003 views
    File comment
    Bochs BIOS in UniPCemu (PureVGA)
    File license
    Public domain

Reply 43 of 58, by superfury

User metadata
Rank l33t++
Rank
l33t++
mr.cat wrote on 2021-07-05, 10:09:
Right, right. Yes that would be the "more extended" part of the thread title, thx for spelling it out in a bit more detail. […]
Show full quote

Right, right. Yes that would be the "more extended" part of the thread title, thx for spelling it out in a bit more detail.

Btw the exe package is there but the name is in lower case, it's just the link that's broken:
http://www.rcollins.org/ftp/dloads/vme1.zip

EDIT: Since I've been playing around with Bochs BIOS a bit, here's a little follow-up on that SeaBIOS commentary on the previous page...
The "legacy" variant does try to boot with i440FX+PureVGA, but I had to cheat a bit. It hangs at POST20 unless that part is commented out.
Note that VGAROM.BIN must also be one of Bochs' variants.

EDIT2: Yay, got the CD-ROM boot going by manually overriding the "bootfirst" variable (see rombios.c). Looks like plop is not necessary with this kind of setup.
Don't wanna jinx it, but I'm sure the trials and tribulations don't stop there...

Do you mean the POST port reports 20h? That's in the middle of the PS/2 output (port 60h) waiting for status bit 2 to clear for command 60h(line 1938)?
That one should have been emulated properly though?

So looking at the POST code with the legacy BIOS, it sends 0xFF to the PS/2 keyboard, waits for it to send, then times out waiting for the ACK to come from the PS/2 keyboard?

Edit: OK. The keyboard reset part is an issue on the Bochs BIOS itself:
https://sourceforge.net/p/bochs/code/HEAD/tre … rombios.c#l1884

It writes the two PS/2 port enable commands, immediately followed by the command to the PS/2 keyboard itself. But it doesn't properly wait for bit 2 of the status to clear.
That's line 1867-1870 needs to be pasted after each of those two writes(after line https://sourceforge.net/p/bochs/code/HEAD/tre … rombios.c#l1885 and https://sourceforge.net/p/bochs/code/HEAD/tre … /rombios.c#1886).

Edit: OK. After those, there was one little Bochs-related issue left, but this time inside UniPCemu. It seems Bochs' sending of a keyboard reset expects a fast BAT to occur (not the 600ms UniPCemu uses). Changing it to a fast response (100us) made the Bochs BIOS continue POSTing past the timing issue without issues (as long as the fix in the rom bios is present).

I've made a little upload of the Bochs BIOS I've used (just changed the code in biosrom.c for Bochs, then ran ../configure in the bios directory, followed by "make bios-bochs-legacy" to build the ROM):

Filename
BIOSROM.i440fx_bochslegacypatch.zip
File size
23.62 KiB
Downloads
45 downloads
File comment
Bochs legacy ROM with my init_keyboard bugfix added.
File license
Fair use/fair dealing exception

Floppy booting seems to work?

Author of the UniPCemu emulator.
UniPCemu Git repository
UniPCemu for Android, Windows, PSP, Vita and Switch on itch.io

Reply 44 of 58, by mr.cat

User metadata
Rank Member
Rank
Member

Thanks, that's the one. Adding those wait loops fixed the POST20 hang. Here's the patch I'm using:

$ diff rombios.c.original rombios.c
1868a1869,1873
> /* Wait until buffer is empty */
> max=0xffff;
> while ((inb(PORT_PS2_STATUS) & 0x02) && (--max>0)) outb(PORT_DIAG, 0x10);
> if (max==0x0) keyboard_panic(10);
>
1869a1875,1878
> /* Wait until buffer is empty */
> max=0xffff;
> while ((inb(PORT_PS2_STATUS) & 0x02) && (--max>0)) outb(PORT_DIAG, 0x10);
> if (max==0x0) keyboard_panic(10);
8308c8317,8318
< bootfirst = read_word(IPL_SEG, IPL_BOOTFIRST_OFFSET);
---
> /*bootfirst = read_word(IPL_SEG, IPL_BOOTFIRST_OFFSET);*/
> bootfirst = 3; /* UniPCemu testing: Force first cdrom */

...and yeah, using that debuggerqemu cli parameter is highly recommended if there's a need to do any problem solving with this BIOS.
Actually I'm not sure about floppy booting (only tried with cdrom so far). EDIT: Tested it, wouldn't work & reasons seem to be explained below...
Hmm also having some kbd issues but this could actually be a Linux specific thing. Needs some testing I suppose...
EDIT2: Never mind, got that sorted (user error)

Last edited by mr.cat on 2021-07-06, 18:17. Edited 2 times in total.

Reply 45 of 58, by superfury

User metadata
Rank l33t++
Rank
l33t++

Well, regarding that CD-ROM boot... Floppy boot seems to run, but when I try to boot the CD-ROM using the boot menu, it somehow disables the keyboard after receiving the F12 key somehow?

Edit: Managed to fix that input issue by fixing the int09 handler in much the same way:

;-------------------------------------------------
;- INT09h : Keyboard Hardware Service Entry Point -
;-------------------------------------------------
.org 0xe987
int09_handler:
cli
push ax

mov al, #0xAD ;;disable keyboard
out PORT_PS2_STATUS, al
loop_int09_handler_disable:
in al,PORT_PS2_STATUS
test al,#0x02
jnz loop_int09_handler_disable


mov al, #0x0B
out PORT_PIC1_CMD, al
in al, PORT_PIC1_CMD
and al, #0x02
jz int09_finish

in al, PORT_PS2_DATA ;;read key from keyboard controller
sti
push ds
pusha
#ifdef BX_CALL_INT15_4F
mov ah, #0x4f ;; allow for keyboard intercept
stc
int #0x15
push bp
mov bp, sp
mov [bp + 0x10], al
pop bp
jnc int09_done
#endif

;; check for extended key
push #0x40
pop ds
cmp al, #0xe0
jne int09_check_pause
mov al, BYTE [0x96] ;; mf2_state |= 0x02
or al, #0x02
mov BYTE [0x96], al
jmp int09_done

int09_check_pause: ;; check for pause key
cmp al, #0xe1
jne int09_process_key
mov al, BYTE [0x96] ;; mf2_state |= 0x01
or al, #0x01
mov BYTE [0x96], al
jmp int09_done

int09_process_key:
call _int09_function

int09_done:
popa
Show last 17 lines
  pop   ds
cli
call eoi_master_pic

;; Notify keyboard interrupt complete w/ int 15h, function AX=9102
mov ax, #0x9102
int #0x15

int09_finish:
mov al, #0xAE ;;enable keyboard
out PORT_PS2_STATUS, al
loop_int09_handler_enable:
in al, PORT_PS2_STATUS
test al, #0x02
jnz loop_int09_handler_enable
pop ax
iret
Filename
BIOSROM.i440fx_bochslegacypatch-int09fix.zip
File size
23.63 KiB
Downloads
42 downloads
File comment
The same POST fix with the interrupt handler fix added in as well.
File license
Fair use/fair dealing exception
Filename
rombios.c.patch
File size
659 Bytes
Downloads
56 downloads
File comment
ROMBIOS patch
File license
GPL-2.0-or-later

Although I'm not quite sure if the file license of GNU 2.0 or later is proper for the Bochs LGPL license for the patch.

Edit: Looking at the FDC code inside the Bochs BIOS, it looks like it doesn't perform anything like stuff like status register checks as well? So that would fail on a real FDC as well?

I won't be that surprised anymore by now if there's much of the code that's actually compatible with a real (fully emulated) x86 system at all? Looking at all the shortcuts it and the Bochs emulator takes (in a sense worse than (vanilla) Dosbox afaik).
Mostly stuff like timings and hardware buffers and atsyusrs just adsumed and not taken into account (much like the way of it's 8042 emulation and handling I mentioned here.
It;s not that the FDC is faulty, it's the BIOS that's very incomplete there, skipping lots of necessary checks required for real hardware!

Author of the UniPCemu emulator.
UniPCemu Git repository
UniPCemu for Android, Windows, PSP, Vita and Switch on itch.io

Reply 46 of 58, by mr.cat

User metadata
Rank Member
Rank
Member

That makes sense. The relations between these projects aren't very clear to me, but I assume there to be differences simply because of different targets:
SeaBIOS (and coreboot) targets real machines, while Bochs doesn't, so it can take those shortcuts.
Just a hunch, but they've probably ripped out everything that isn't strictly needed to get Bochs up and running.
Note that the only reason I took that for a spin was because it was the easiest one to test. The Bochs legacy one also ended up being the only variant I got to boot this far ¯\_(ツ)_/¯
The SeaBIOS/coreboot ones would definitely be interesting, but from user pov it doesn't matter which exact variant is used - what matters is usability.

I wouldn't worry about the patch licenses too much. The original author always has their say about that and using multiple licenses is also possible.
You can't change the license retrospectively though, yeah could be problematic in some cases but I'd say mostly it's not an issue.

Reply 47 of 58, by superfury

User metadata
Rank l33t++
Rank
l33t++

I've added some (Visual Studio only) opcode jump tables that show the jumptables in a more viewable format (instead of being formatted opcode at bits 2+, 0F at bit 1 and 16/32-bit at bit 0, it's the same, but in a more readable format to compare to http://ref.x86asm.net/coder32.html). So the table it creates in that case is a multidimensional one with 3 levels, addressed like this: table[0F][opcodebyte][bitness], where 0F=0/1, opcodebyte=00-FF, bitness=0(16)/1(32-bit operand size).

Having looked through all those opcodes, it matches the coder32 edition perfectly.
So the opcode jumptables as well as the instruction fetching(which was tested using the Baresifter tests) are correct.
I've also tested most of the emulated opcodes, with only a handful that don't seem to be used so far.
So probably, the opcodes and it's parsing isn't the issue here? It's probably a CPU mechanism error itself that's showing itself in failing to boot? Like exception handling, interrupt handling, fault handling, task switches and other related things?

Perhaps the page fault that was fixed was the cause of all issues? But Windows 2000 still can't boot (which is from CD-ROM), so that's still not the issue here? NT 4.0 might need a full reinstallation to be sure though?

Author of the UniPCemu emulator.
UniPCemu Git repository
UniPCemu for Android, Windows, PSP, Vita and Switch on itch.io

Reply 48 of 58, by superfury

User metadata
Rank l33t++
Rank
l33t++
mr.cat wrote on 2021-06-29, 11:07:
OK that's good to know. Dev tools can be compiled from scratch, but that can be a real hassle (especially starting from 1997!) T […]
Show full quote

OK that's good to know. Dev tools can be compiled from scratch, but that can be a real hassle (especially starting from 1997!)
The booting issue could perhaps be resolved by trying out those older ones.

OTOH baresifter sidesteps these issues altogether, so perhaps it'd be a better fit.
Here's a ready-made boot floppy if you want to give it a try. Do note that the output goes to serial port by default.
I renamed baresifter.x86_32.elf to bsifter, so that's the file to replace in case you need to customize.

EDIT: Changed the output device to 0xE9 (see superfury's explanation below).
EDIT2: OK found another stumbling block...
Instruction length tests fail because one of the tests uses the xbegin opcode, so I just removed that from main.cpp.
Strangely enough, that test does pass in qemu (with -cpu pentium2).
EDIT3: Added some DEBUG logging to self tests.
EDIT4: Added NX detection (from git issue).
EDIT5: If you can't boot from floppies (as seems to be the case with Bochs BIOS), you can convert the floppy image to bootable iso, e.g. something like this:

mkdir boot
mv -i /path/to/BSIFTER.IMG boot/.
genisoimage -o BSIFTER.ISO -b BSIFTER.IMG boot

Is it possible to get or generate an i386-compatible ELF-loading image somehow? I managed to port (although untested PDE-PTE creation on i486/i386 due to the ELF boot loader on the image you've provided crashing on those (using i686 opcodes)). The one you supplied is compiled for i686.
That way I can test the new baresifter version I adjusted to be able to run on i386/i486 CPUs (mainly the thing that still needs to be tested is i386-compatible paging right now). Other than actual sifting to find bugs of course.

The i386 changed code for the i386 ELF for baresifter can be found at:
https://github.com/superfury/baresifter/

And the compiled version (on latest Ubuntu Virtualbox VM):

Filename
baresifter.x86_i386.zip
File size
44.41 KiB
Downloads
8 downloads
File comment
The compiled executable with i386-based paging support (as well as i486 and i586 support).
File license
Fair use/fair dealing exception

Edit: Seeing as I can't seem to force qemu to use the older CPUs directly (nor having a valid boot loader right now for i386 CPUs), one thing that i can do is force the CPUID instruction off in a custom build of the source code. That should force it to detect a 486 or older, even though it's actually a 586 or newer (and qemu reporting that it's supporting 4MB pages). Then UniPCemu itself can tell me if it works properly by crashing the app or not (or starting to sift).
OK. It looks like the i386/i486 paging system works properly, as I actually see it performing the sifting without PSE enabled!

So the 32-bit ELF image at least (assuming the compiler doesn't put i686 opcodes in there, which I can't verify with my current setup, as I can only run it on a i686 CPU due to the ELF loader requiring i686 instructions) is compatible with i486 CPUs now. Although some CPUID detection for writing to port E9h doesn't allow it to log anything (other than to the serial port). It also doesn't support the port E9 hack fully (reading port E9h results in the value E9h being read back).
Perhaps adding port E9h reads returning E9h within the source code might help?
Otherwise, I could also simply connect using COM0COM to a simple putty client to view everything written to the serial port using UniPCemu.

Last edited by superfury on 2024-03-11, 21:36. Edited 1 time in total.

Author of the UniPCemu emulator.
UniPCemu Git repository
UniPCemu for Android, Windows, PSP, Vita and Switch on itch.io

Reply 49 of 58, by mr.cat

User metadata
Rank Member
Rank
Member

Hi, that's great to hear.
Unfortunately I have no recollection about how exactly I originally generated the floppy. Must have been with syslinux I guess?
The executable can be updated with mcopy (part of mtools) as described here.

Reply 50 of 58, by superfury

User metadata
Rank l33t++
Rank
l33t++
mr.cat wrote on 2024-03-10, 17:21:

Hi, that's great to hear.
Unfortunately I have no recollection about how exactly I originally generated the floppy. Must have been with syslinux I guess?
The executable can be updated with mcopy (part of mtools) as described here.

The problem on 80386/80486/Pentium isn't the ELF executable (it's properly built for the i386).
In my case I use WinImage to put it on the disk.

The problem is the (boot) loader (SYSLINUX from the filenames and banner?) that loads it uses i686 instructions (HINT NOP at least), thus crashing before or while loading the ELF executable into memory. It displays the banner, then crashes (#UD exception loop) on i586 and lower.

The configuration file contains this:

DEFAULT baresifter
LABEL baresifter
COM32 mboot.c32
APPEND bsifter

Don't know what COM32 or mboot.c32 are or if they're i386-compatible though.

mboot.c32 is probably required to load the ELF file? What about libcom32.c32?
Edit: OK. libcom32.c32 is required by mboot.c32 (https://wiki.syslinux.org/wiki/index.php?titl … Library_modules).

Now I would need all those files in i386 architecture.

Edit: Perhaps Debian Bo can help there?
https://archive.debian.org/debian/dists/bo/ma … nary-i386/base/
It seems to have syslinux as well. Perhaps i386 compatible?

Edit: After about 24 hours, it's about 24 hours more until the baresifter processing is supposed to finish (according to a simple time converting calculating). Right now it's at:

24:35:58:32.09856: EXC 0E OK | 2B 24 B5 00 00 00 00

So it's been sifting for slightly over 1 day already. (CPU at 20000KIPS, 3% realtime speed)

Last edited by superfury on 2024-03-11, 22:17. Edited 2 times in total.

Author of the UniPCemu emulator.
UniPCemu Git repository
UniPCemu for Android, Windows, PSP, Vita and Switch on itch.io

Reply 51 of 58, by superfury

User metadata
Rank l33t++
Rank
l33t++

Just quickly looked at the current dump of instructions measured so far.

I don't see the 66h and 67h prefixes in there? So the testsuite is actively skipping those!
So baresifter is only testing the 32-bit halves of the 16/32-bit instructions!

Edit: Also, on x86, 40-4F is counted as a prefix, which it isn't?
Edit: Just fixed that in my repository (the invalid x86 REX prefixes, compiled version in this post).

So I'll need to pass command line parameters to the executable to set the prefix count to properly cover it all.
Thinking about that, only 2 prefixes actually change the length: the operand and address size prefixes.
So that's requiring 2 prefixes to be set, but it currently uses 0 prefixes, so nothing is checked.

Edit: Is it possible to give command line parameters to the ELF executable? Perhaps by simply adding them to the filename with spaces in syslinux.cfg? So prefixes=2?

Edit: Baresifter with fixed i386 (REX) prefixes.

Filename
baresifter.i386prefixes.zip
File size
44.42 KiB
Downloads
6 downloads
File comment
Baresifter i386 with prefixes fixed.
File license
Fair use/fair dealing exception

Edit: The prefixes=2 parameter seems to work! Now 16-bit instructions should at last be checked too!
Although it would multiply the time taken to sift to a few days at least? Normally about 48 hours (2 days), now probably multiplied by 4, so 8 days to sift? And ofc needs to start all over again, as the algorithm changed.

Author of the UniPCemu emulator.
UniPCemu Git repository
UniPCemu for Android, Windows, PSP, Vita and Switch on itch.io

Reply 52 of 58, by superfury

User metadata
Rank l33t++
Rank
l33t++

I've been thinking a bit.

The analyze program that's included with Baresifter uses some kind of command line parameters:

#[derive(Clap, Debug)]
#[clap(version = crate_version!(), author = "Julian Stecklina <js@alien8.de>")]
struct Opts {
/// The address width to use.
#[clap(long, default_value = "64")]
bits: u8,

/// The input file to parse.
input_file: PathBuf,
}

How do I specify those two parameters without changing the source code? Do I simply add "32 <pathtoporte9logfile>" to the command line of the executable? Or do I need to use cargo for that somehow?

Is that simply:
"cargo run -- 32 <pathtoporte9logfile>"?

In my case the directory of the baresifter repository is inside a Ubuntu VM (Virtualbox) inside "~/baresifter/baresifter". And the UniPCemu logs directory is at "/media/sf_projects/projects_build/UniPCemu/logs".
So running it against the log would be:
"cargo run -- 32 /media/sf_projects/projects_build/UniPCemu/logs/porte9.log"

Is that the correct way to disassemble 32-bit?

Last edited by superfury on 2024-03-12, 14:51. Edited 1 time in total.

Author of the UniPCemu emulator.
UniPCemu Git repository
UniPCemu for Android, Windows, PSP, Vita and Switch on itch.io

Reply 53 of 58, by mr.cat

User metadata
Rank Member
Rank
Member

Yeah seems legit, the double dash is used to signify the arguments that belong to the target executable.
cargo run has the side-effect that it will try to recompile the target if it deems it necessary.
If you don't like that, you can use the exe directly (it resides somewhere in the target subdirectory)

Reply 54 of 58, by superfury

User metadata
Rank l33t++
Rank
l33t++
mr.cat wrote on 2024-03-12, 13:28:

Yeah seems legit, the double dash is used to signify the arguments that belong to the target executable.
cargo run has the side-effect that it will try to recompile the target if it deems it necessary.
If you don't like that, you can use the exe directly (it resides somewhere in the target subdirectory)

That's what I did before, just calling the generated executable directly.
Although I modified the source code to use the 32-bit parameter instead of using a command line parameter for that.
Any idea what the exact parameter in that case is? The filename is obviously just entered normally. How would I specify the 32-bit setting?

Edit: OK. Built it with "cargo build".

Then went to the next step and tried getting some help:

cargo run -- --help

Which gave me:

baresifter-analyze 0.1.0



Julian Stecklina <js@alien8.de>



USAGE:

baresifter-analyze [OPTIONS] <INPUT_FILE>



ARGS:

<INPUT_FILE> The input file to parse



FLAGS:

-h, --help Print help information

-V, --version Print version information



OPTIONS:

--bits <BITS> The address width to use [default: 64]

Yay, I can pass arguments!

So basically, it would be:

cargo run -- --bits 32 <pathtologfile>

So to parse in 32-bit UniPCemu's folder containing the "projects_build" folder output mounted at "projects" (turned into /media/sf_projects inside the VM) inside Virtualbox:

cargo run -- --bits 32 /media/sf_projects/projects_build/UniPCemu/logs/porte9.log

That should parse the output correctly?
Edit: Right now it's sifting at "17:55:33:82.04512: EXC 0E OK | 12 3C 25 00 00 00 00".

Author of the UniPCemu emulator.
UniPCemu Git repository
UniPCemu for Android, Windows, PSP, Vita and Switch on itch.io

Reply 55 of 58, by superfury

User metadata
Rank l33t++
Rank
l33t++

Just have been thinking. About the #UD opcodes, what happens on the paging fetching of the following opcodes on 386 till Pentium 2?

0F FF: does this have a modr/m? UniPCemu doesn't have any.
0F B9: does this have a 16-bit modr/m? the documentation I can find on it only mentions 32-bit modr/m. UniPCemu implements no modr/m fetching at all for 80286+ (so all protected-mode CPUs it emulates). So does it have no opcode, modr/m32(like MOV CR0) or modrm16/32(like most 16-bit instructions)?

Edit: Current progress:

22:21:33:77.08944: EXC 01 OK | 26 0F 0D 14 75 00 00 00 00

Author of the UniPCemu emulator.
UniPCemu Git repository
UniPCemu for Android, Windows, PSP, Vita and Switch on itch.io

Reply 56 of 58, by superfury

User metadata
Rank l33t++
Rank
l33t++

Improved the Baresifter program a bit.
Added 2 bitmask parameters: detect_prefixes and used_prefixes.
Both default to 0xFF (use all available) for compatibility.

Now the two settings work as follows:
- The bit in the variable represents the prefix group (bit 0=LOCK/REPNE/REP, bit 1=segment overrides, bit 2=operand size override, bit 3=address size override, bit 4=REX prefixes).
- Clearing a bit excludes a group from detection or being used in sifting (not being used effectively skips the instructions containing them).

detect_prefixes affects how the prefixes are detected at all. Clearing a bit makes it behave like an opcode, otherwise it's a prefix.
This thus affects whether a prefix is a prefix or an opcode.

Then, once the prefix vs opcode and count is determined, when filtering the instructions to execute, clearing the bit in used_prefixes filters it to not appear in the results (effectively being skipped from running).

The nice thing about is, you can combine the two to disable certain prefixes from being prefixes on certain machines (like REX not existing on a pure x86 machine, being a normal opcode instead), thus detect_prefixes is required.
On plain x86 (until Pentium 4 at least, probably newer as well) you'd set detect_prefixes to 15, thus excluding REX from being a prefix (it's actually an INC/DEC opcode instead).

Then, when sifting, you might want to exclude prefixes that don't affect the instruction in a visible way anyway (other than faults perhaps). Thus you can exclude the segment overrides at least (thus setting used_prefixes=13 on x86 machines).

So this is what I end up with right now:

Filename
baresifter_floppy_operand-addresssize_lock-repne-rep.zip
File size
310.24 KiB
Downloads
5 downloads
File comment
Baresifter on floppy, latest build, 2 prefixes, operand/address size, lock/rep/repne prefixes enabled only in the syslinux.cfg
File license
Fair use/fair dealing exception
Filename
baresifter.x86_i386_prefixesconfigurable.zip
File size
45.98 KiB
Downloads
3 downloads
File comment
Baresifter compiled with configurable prefixes
File license
Fair use/fair dealing exception

Edit: Although the Analyzer that's included can't skip opcode groups yet so far. Haven't looked at that one.
UniPCemu is now happily churning away at that latest baresifter build, with 2 prefixes etc. and the other prefixes (REX detection disabled and segment override prefixes skipped).
That will cause the segment override instructions to error out, but will provide much better coverage because of the reducing of executed instructions inside UniPCemu, which is slow enough as it is.

Edit: Actually, screw that. Excluding LOCK/REP/REPNZ as well, as those only affect the faulting behaviour in this case, not the instruction length I want to know about.
So restarting once again.

Filename
baresifter_floppy_opeandaddresssizeonly.zip
File size
310.24 KiB
Downloads
4 downloads
File comment
Baresifter on floppy, latest build, 2 prefixes, operand and address size prefixes only.
File license
Fair use/fair dealing exception

(Although all that's changed is the syslinux.cfg file in this case)

Simply the address and operand size prefixes (not using any of those other prefixes) would make the Analyzer spew out all those opcodes using that, but the instructions not using those prefixes should be relatively easy to spot in the error log?
Or perhaps some option is needed to exclude those from the analyzer as well somehow?

Author of the UniPCemu emulator.
UniPCemu Git repository
UniPCemu for Android, Windows, PSP, Vita and Switch on itch.io

Reply 57 of 58, by superfury

User metadata
Rank l33t++
Rank
l33t++

Anyone knows if it's possible to get the search engine's filtered prefix detection with skipping to work with the Baresifter analyzer program?
Right now it would give a lot of issues with the skipped and undetected prefixes I think (results are still running inside UniPCemu for a few days at least)?

Till roughly 0FC0 in about 12 hours I think. So not fast unfortunately. That's roughly 25%? 48 hours in total roughly for the entire range? Times 4 for adding operand and address prefixes? So ~120 hours or 4 days to sift through?

Ofc that's with most prefixes (CS/SS/DS/ES/FS/GS and LOCK/REP/REPNE) disabled, since they don't affect instruction length (only faults and special effects of the instruction and used segments (CS/DS also optimization, but that's unimplemented, effectively more something like the likely() functionality)).

Edit: Current progress...

10:57:14:44.06848: EXC 0E OK | 0F B3 04 55 00 00 00 00

Author of the UniPCemu emulator.
UniPCemu Git repository
UniPCemu for Android, Windows, PSP, Vita and Switch on itch.io

Reply 58 of 58, by superfury

User metadata
Rank l33t++
Rank
l33t++
57:15:11:45.08304: >>> Done!

It's finally finished. So 2 days 9 hours and 15:11:45.08304 minutes to complete. So roughly 2.3 days to complete.

Results:

Filename
porte9_baresifter_20240314_0939.zip
File size
1.33 MiB
Downloads
3 downloads
File comment
Finished baresifter output.
File license
Fair use/fair dealing exception

The command for the analyzer is almost correct:
It's supposed to be:

cargo run -- --bits 32 <path to porte9.log file>

Edit: Nonetheless, interesting output:

26 00 00 00 00 00 00 00 00 00 00 00 00 00 00 | 05 ≠ 03 add es:[eax],al

2e 00 00 00 00 00 00 00 00 00 00 00 00 00 00 | 05 ≠ 03 add cs:[eax],al

36 00 00 00 00 00 00 00 00 00 00 00 00 00 00 | 05 ≠ 03 add ss:[eax],al

3e 00 00 00 00 00 00 00 00 00 00 00 00 00 00 | 05 ≠ 03 add [eax],al

64 00 00 00 00 00 00 00 00 00 00 00 00 00 00 | 02 ≠ 03 add fs:[eax],al

65 00 00 00 00 00 00 00 00 00 00 00 00 00 00 | 02 ≠ 03 add gs:[eax],al

66 64 00 00 00 00 00 00 00 00 00 00 00 00 00 | 03 ≠ 04 add fs:[eax],al

66 65 00 00 00 00 00 00 00 00 00 00 00 00 00 | 03 ≠ 04 add gs:[eax],al

66 66 00 00 00 00 00 00 00 00 00 00 00 00 00 | 03 ≠ 04 add [eax],al

66 67 64 00 00 00 00 00 00 00 00 00 00 00 00 | 04 ≠ 05 add fs:[bx+si],al

66 67 65 00 00 00 00 00 00 00 00 00 00 00 00 | 04 ≠ 05 add gs:[bx+si],al

66 67 66 00 00 00 00 00 00 00 00 00 00 00 00 | 04 ≠ 05 add [bx+si],al

66 67 67 00 00 00 00 00 00 00 00 00 00 00 00 | 04 ≠ 05 add [bx+si],al

66 67 f0 00 00 00 00 00 00 00 00 00 00 00 00 | 03 ≠ 05 lock add [bx+si],al

66 67 f2 00 00 00 00 00 00 00 00 00 00 00 00 | 03 ≠ 05 add [bx+si],al

66 67 f3 00 00 00 00 00 00 00 00 00 00 00 00 | 03 ≠ 05 add [bx+si],al

66 f0 00 00 00 00 00 00 00 00 00 00 00 00 00 | 02 ≠ 04 lock add [eax],al

66 f2 00 00 00 00 00 00 00 00 00 00 00 00 00 | 02 ≠ 04 add [eax],al

66 f3 00 00 00 00 00 00 00 00 00 00 00 00 00 | 02 ≠ 04 add [eax],al

67 26 00 00 00 00 00 00 00 00 00 00 00 00 00 | 06 ≠ 04 add es:[bx+si],al

67 2e 00 00 00 00 00 00 00 00 00 00 00 00 00 | 06 ≠ 04 add cs:[bx+si],al

67 36 00 00 00 00 00 00 00 00 00 00 00 00 00 | 06 ≠ 04 add ss:[bx+si],al

67 3e 00 00 00 00 00 00 00 00 00 00 00 00 00 | 06 ≠ 04 add [bx+si],al

67 64 00 00 00 00 00 00 00 00 00 00 00 00 00 | 03 ≠ 04 add fs:[bx+si],al

67 65 00 00 00 00 00 00 00 00 00 00 00 00 00 | 03 ≠ 04 add gs:[bx+si],al

67 66 00 00 00 00 00 00 00 00 00 00 00 00 00 | 03 ≠ 04 add [bx+si],al

67 67 00 00 00 00 00 00 00 00 00 00 00 00 00 | 03 ≠ 04 add [bx+si],al

67 f0 00 00 00 00 00 00 00 00 00 00 00 00 00 | 02 ≠ 04 lock add [bx+si],al

67 f2 00 00 00 00 00 00 00 00 00 00 00 00 00 | 02 ≠ 04 add [bx+si],al

67 f3 00 00 00 00 00 00 00 00 00 00 00 00 00 | 02 ≠ 04 add [bx+si],al

Show last 5 lines
f0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 | 01 ≠ 03 lock add [eax],al

f2 00 00 00 00 00 00 00 00 00 00 00 00 00 00 | 01 ≠ 03 add [eax],al

f3 00 00 00 00 00 00 00 00 00 00 00 00 00 00 | 01 ≠ 03 add [eax],al

Ignoring the unused prefixes, there's nothing weird left? So that means that at least the instruction fetching works correctly?
Edit: Looking closer, I see some interesting ones though (filtering the above):

66 67 66 00 00 00 00 00 00 00 00 00 00 00 00 | 04 ≠ 05 add [bx+si],al

66 67 67 00 00 00 00 00 00 00 00 00 00 00 00 | 04 ≠ 05 add [bx+si],al

67 66 00 00 00 00 00 00 00 00 00 00 00 00 00 | 03 ≠ 04 add [bx+si],al

67 67 00 00 00 00 00 00 00 00 00 00 00 00 00 | 03 ≠ 04 add [bx+si],al

Or is that because duplicate prefixes are also filtered out in the baresifter program?

Author of the UniPCemu emulator.
UniPCemu Git repository
UniPCemu for Android, Windows, PSP, Vita and Switch on itch.io