Perhaps Intel was just too ashamed to document the inner workings in enough detail 😀

Question. Now I am also emulating cycle type (T0, T1, T2, T3 and Twait) and I notice gaps in the EU. Say the previous instruction finished on T0, the EU will wait until next T3 before decoding the next instruction (as T3 is when that happens). That is acceptable right? In the trace below look at cycle 37 and 44.

1Cycle  1 T0
2Cycle  2 T1
3Cycle  3 T2
4Cycle  4 T3     prefetch (1/4) 
5Cycle  5 T0
6Cycle  6 T1
7Cycle  7 T2						 new opcode 0xea
8Cycle  8 T3     prefetch (1/4) 	 waiting for data 
9Cycle  9 T0						 waiting for data 
10Cycle 10 T1						 waiting for data 
11Cycle 11 T2						 waiting for data 
12Cycle 12 T3     prefetch (2/4) 	 waiting for data 
13Cycle 13 T0						 waiting for data 
14Cycle 14 T1						 waiting for data 
15Cycle 15 T2						 waiting for data 
16Cycle 16 T3     prefetch (1/4) 	 waiting for data 
17Cycle 17 T0						 waiting for data 
18Cycle 18 T1						 waiting for data 
19Cycle 19 T2						 waiting for data 
20Cycle 20 T3     prefetch (2/4) 	 waiting for data JMP 0xf000:0xe05b
21Cycle 21 T0						 instruction time 
22Cycle 22 T1						 instruction time 
23Cycle 23 T2						 instruction time 
24Cycle 24 T3     prefetch (1/4) 	 instruction time 
25Cycle 25 T0						 instruction time 
26Cycle 26 T1						 instruction time 
27Cycle 27 T2						 instruction time 
28Cycle 28 T3     prefetch (2/4) 	 instruction time 
29Cycle 29 T0						 instruction time 
30Cycle 30 T1						 instruction time 
31Cycle 31 T2						 instruction time 
32Cycle 32 T3     prefetch (3/4) 	 instruction time 
33Cycle 33 T0						 instruction time 
34Cycle 34 T1						 instruction time 
35Cycle 35 T2						 instruction time 
36Cycle 36 T3     prefetch (4/4) 	 instruction time 
37Cycle 37 T0
38Cycle 38 T1
39Cycle 39 T2						 new opcode 0xb8 MOV AX, 0x40
40Cycle 40 T3     prefetch (2/4) 	 instruction time 
41Cycle 41 T0						 instruction time 
42Cycle 42 T1						 instruction time 
43Cycle 43 T2						 instruction time 
44Cycle 44 T3     prefetch (3/4) 
45Cycle 45 T0
46Cycle 46 T1
47Cycle 47 T2						 new opcode 0x8e MOV DS, AX
48Cycle 48 T3     prefetch (2/4) 	 instruction time 
49Cycle 49 T0						 instruction time 
50Cycle 50 T1
51Cycle 51 T2						 new opcode 0xc7

The four bus cycles are T1, T2, T3, T4 not T0, T1, T2, T3 (I usually prefer to count starting from 0 too but in this case I'm just following the nomenclature in Intel's documentation).

"When instruction decoding happens" is not something that can be observed or deduced - all we can see is the signals on the CPU pins (bus and queue status). So the closest we can get is the "I" signal on the queue status pins (opcode grabbed from prefetch queue) or the last "S" signal for the instruction (which is the line where I show the instruction being executed in my dumps, for the sake of convenience in writing the decoder). In general, either of these can happen on any bus T-state. In the specific case of a 5-byte instruction and an initially-empty or initially-full prefetch queue, the 5th byte will be grabbed from the prefetch queue on a T3 cycle because the delays involved in putting the prefetched byte into the queue (3 cycles after the T4 of the prefetch of the 5th byte).

For the specific instructions you're emulating here (initially empty prefetch queue, "JMP cp / MOV AX, iw / MOV DS, AX") here is what my sniffer reports:

104ABF .C...  00A2B EA 00 FC 4 .......  T1
200F2B .C...  00F2B EA 00 FC 4 .......  T2
320F2B .C...  00F2B FF 00 FC 5 ..r....  T3
420FEA .p...  00F2B EA 00 FC 5 ..r....  T4    EA <-f [   00F2B]
520FEA .C...  00F2B EA 00 FC 4 .......  T1
600F2C .C...  00F2C EA 00 FC 4 .......  T2
720F2C IC...  00F2C FF 00 FC 5 ..r....  T3                      I
820FB0 .p...  00F2C B0 00 FC 5 ..r....  T4    B0 <-f [   00F2C]
920FB0 .C...  00F2C B0 00 FC 4 .......  T1
1000F2D .C...  00F2D B0 00 FC 4 .......  T2
1120F2D SC...  00F2D FF 00 FC 5 ..r....  T3                      S
1220F04 .p...  00F2D 04 00 FC 5 ..r....  T4    04 <-f [   00F2D]
1320F04 .C...  00F2D 04 00 FC 4 .......  T1
1400F2E .C...  00F2E 04 00 FC 4 .......  T2
1520F2E SC...  00F2E FF 00 FC 5 ..r....  T3                      S
1620FA8 .p...  00F2E A8 00 FC 5 ..r....  T4    A8 <-f [   00F2E]
1720FA8 .C...  00F2E A8 00 FC 4 .......  T1
1800F2F .C...  00F2F A8 00 FC 4 .......  T2
1920F2F SC...  00F2F FF 00 FC 5 ..r....  T3                      S
2020F00 .p...  00F2F 00 00 FC 5 ..r....  T4    00 <-f [   00F2F]
2120F00 .C...  00F2F 00 00 FC 4 .......  T1
2200F30 .C...  00F30 00 00 FC 4 .......  T2
2320F30 SC...  00F30 FD 00 FC 5 ..r....  T3                      S EAB004A800   JMP 00A8:04B0
2420FB8 .p...  00F30 B8 00 FC 5 ..r....  T4    B8 <-f [   00F30]
2520FB8 .p...  00F30 B8 00 FC 4 .......
2620FB8 .p...  00F30 B8 00 FC 4 .......
2720FB8 .p...  00F30 B8 00 FC 5 .......
2820FB8 Ep...  00F30 B8 00 FC 5 .......                          E
2920FB8 .C...  00F30 B8 00 FC 4 .......  T1
3000F30 .C...  00F30 B8 00 FC 4 .......  T2
3120F30 .C...  00F30 FF 00 FC 5 ..r....  T3
3220FB8 .p...  00F30 B8 00 FC 5 ..r....  T4    B8 <-f [   00F30]
3320FB8 .C...  00F30 B8 00 FC 4 .......  T1
3400F31 .C...  00F31 B8 00 FC 4 .......  T2
3520F31 IC...  00F31 FF 00 FC 5 ..r....  T3                      I
3620F40 .p...  00F31 40 00 FC 5 ..r....  T4    40 <-f [   00F31]
3720F40 .C...  00F31 40 00 FC 4 .......  T1
3800F32 .C...  00F32 40 00 FC 4 .......  T2
3920F32 SC...  00F32 FF 00 FC 5 ..r....  T3                      S
4020F00 .p...  00F32 00 00 FC 5 ..r....  T4    00 <-f [   00F32]
4120F00 .C...  00F32 00 00 FC 4 .......  T1
4200F33 .C...  00F33 00 00 FC 4 .......  T2
4320F33 SC...  00F33 FD 00 FC 5 ..r....  T3                      S B84000       MOV AX, 0040
4420F8E .p...  00F33 8E 00 FC 5 ..r....  T4    8E <-f [   00F33]
4520F8E .C...  00F33 8E 00 FC 4 .......  T1
4600F34 .C...  00F34 8E 00 FC 4 .......  T2
4720F34 IC...  00F34 FF 00 FC 5 ..r....  T3                      I
4820FD8 .p...  00F34 D8 00 FC 5 ..r....  T4    D8 <-f [   00F34]
4920FD8 .C...  00F34 D8 00 FC 4 .......  T1
5000F35 .C...  00F35 D8 00 FC 4 .......  T2
5120F35 SC...  00F35 FF 00 FC 5 ..r....  T3                      S 8ED8         MOV DS, AX
5220FC3 .p...  00F35 C3 00 FC 5 ..r....  T4    C3 <-f [   00F35]
5320FC3 .C...  00F35 C3 00 FC 4 .......  T1
5400F36 .C...  00F36 C3 00 FC 4 .......  T2
5520F36 IC...  00F36 FF 00 FC 5 ..r....  T3                      I C3           RET

It doesn't seem to quite match up with your emulation yet. In particular, the CPU turns off prefetching at some point during the execution of the JMP instruction (so only one byte after that instruction is prefetched before the queue is emptied). The cycles where the instruction is displayed are offset between your trace and mine by 3, 4 and 4 cycles respectively so I think there's some inconsistency there.

As for "are there gaps in the EU?" - I can't think of any reason why the EU would ever be doing anything other than executing an instruction or waiting for the BIU (assuming you count being stopped in a HLT instruction while waiting for an interrupt as the former). It seems like these two cases are covered by your "waiting for data" and "instruction time" lines, so I'm not sure what the blank third status would mean.

Btw, I think it would help if you added some more information to your dumps - specifically, for each bus transfer, what byte was transferred and which address (IO port, memory or interrupt response) it was transferred to/from. Currently I can't tell if your emulated bus is accessing the correct addresses on the correct cycles.

reenigne wrote:

The four bus cycles are T1, T2, T3, T4 not T0, T1, T2, T3 (I usually prefer to count starting from 0 too but in this case I'm just following the nomenclature in Intel's documentation).

Renamed mine to that.

reenigne wrote:

In general, either of these can happen on any bus T-state.

The reason I mentioned T3 is that I noticed in all your traces (even the last one) that "I" happens on T3 exclusively. But that might be the side-effect of the "tempo" of the EU/bus.

reenigne wrote:

For the specific instructions you're emulating here (initially empty prefetch queue, "JMP cp / MOV AX, iw / MOV DS, AX") here is what my sniffer reports

Thank you for that. If you do not mind, I would like to take you on the offer to use your server to upload my code so I can tune my emulator. I do actually own a 5150 and a 5160 (both working) with a CGA card and a CGA monitor so one day I would like to duplicate your setup to grab my own dump. But that is a topic for a separate thread (and even a separate forum, vcfed.org)

reenigne wrote:

In particular, the CPU turns off prefetching at some point during the execution of the JMP instruction (so only one byte after that instruction is prefetched before the queue is emptied).

That I did not know, I will try to emulate that too.

reenigne wrote:

Btw, I think it would help if you added some more information to your dumps - specifically, for each bus transfer, what byte was transferred and which address (IO port, memory or interrupt response) it was transferred to/from. Currently I can't tell if your emulated bus is accessing the correct addresses on the correct cycles.

Yes, I'll augment my log output.

FWIW, I'll put my code somewhere where people can see it, but lets just say that I work for the biggest company in the world, and I need to navigate some legal waters and make sure I do not upset anyone by putting my own code up. Once I have something that works I'll open my bitbucket repository. I want to make it all open source anyway.

Reenigne, I'm just curious: How did you manage to create that server with the sniffer (without it being able to crash)? Does it store some code made for it in a ROM or harddisk? Does the sniffer use some kind if I/O inferface(I/O ports) to start logging/stop logging/retrieve data logs? Also, what happens when people try to execute breaking stuff like infinite loops (JMP $-2 for example)? Will the sniffer eventually interfere to stop the program or prevent it's execution in another way?

vladstamate wrote:

The reason I mentioned T3 is that I noticed in all your traces (even the last one) that "I" happens on T3 exclusively. But that might be the side-effect of the "tempo" of the EU/bus.

Yes, exactly. In that last trace, the prefetch queue is essentially always empty so the EU ends up synchronized with the BIU but I have other traces where that doesn't happen.

vladstamate wrote:

Thank you for that. If you do not mind, I would like to take you on the offer to use your server to upload my code so I can tune my emulator.

Go ahead. I have noticed in the past couple of days that sometimes it gets itself into a state where it doesn't return information from all the iterations so if you see things like "Warning: missing sniffer item 3" ignore that trace and try again (and let me know if it happens repeatedly). I'm not sure why it happens yet - I've gone over all the code and I don't see the problem so I'm thinking it might be a hardware issue with the serial connection (but that doesn't explain why it always seems to start happening with item 3).

vladstamate wrote:

I do actually own a 5150 and a 5160 (both working) with a CGA card and a CGA monitor so one day I would like to duplicate your setup to grab my own dump.

I do actually have a spare sniffer PCB (the fab company sent me a second one). PM me your snail-mail address if you want it (you'll have to order the rest of the parts and solder it together yourself).

vladstamate wrote:

FWIW, I'll put my code somewhere where people can see it, but lets just say that I work for the biggest company in the world, and I need to navigate some legal waters and make sure I do not upset anyone by putting my own code up. Once I have something that works I'll open my bitbucket repository. I want to make it all open source anyway.

That would be great. I've been working on an emulator as well but it's been pretty slow going so you might very well get yours finished first. If so I think there are quite a few existing PC emulators that would like to be able to use that code.

superfury wrote:

Reenigne, I'm just curious: How did you manage to create that server with the sniffer (without it being able to crash)? Does it store some code made for it in a ROM or harddisk? Does the sniffer use some kind if I/O inferface(I/O ports) to start logging/stop logging/retrieve data logs? Also, what happens when people try to execute breaking stuff like infinite loops (JMP $-2 for example)? Will the sniffer eventually interfere to stop the program or prevent it's execution in another way?

The XT is controlled from a modern (Windows) PC via a microcontroller which is able to do a hard-reset of the XT (by pulling the "power good" line from the power supply low) and load code through the keyboard port using the BIOS's factory test mode (which makes it very quick to start-up, no need to test 640kB of RAM each time). The XT itself doesn't have a hard disk, and is bootstrapped using a program stored in the microcontroller's flash (not accessible from the XT).

The sniffer is completely passive and undetectable from software running on the XT. It monitors some address lines so that it can be controlled from the XT by a program accessing physical addresses 0x80000 (reset sniffer), 0x80008 (send 0 bit to sniffer) and 0x80088 (send 1 bit to sniffer). The trace program uses these commands to send a 40-bit packet to the sniffer to tell it when to start sniffing, how long to start sniffing for and which lines to sniff. Once that sniffer run is complete the sniffer sends the data it captured through a serial connection to the Windows PC, which collects the data from all the runs, reformats it into a text file and puts it on the web server for download. If another program accidentally triggers the sniffer the Windows machine will just ignore that data and the sniffer will complete its cycle in no more than about 200ms.

If you put an infinite loop in the trace program you won't get a proper sniffer dump, but the server process will time-out after 5 minutes and will reset the XT allowing another program to be loaded.

@reenigne: Ok. One strange thing though: you said that it's bootstrapped using a program in the microcontroller's flash, which isn't accessable by the XT. Then what is that program even doing, if anything at all (sounds like a ROM, which is written to, but unreadable by the CPU, thus making the write to it useless to perform at all)? If the code to be executed isn't accessable by the CPU in any way, how can you even run it?

@vladstamate: I've tried the AMD code XL on my Intel i7@4.0Ghz PC, but after profiling I end up with external calls(SDL and Windows) function names, but x86emu.exe only reports addresses? I've entered the paths of the project .c files in the source paths list, but that doesn't help? It is compiled without the profiler flags though(which are, according to what I read, in the case of MinGW, still a WIP to be implemented in AMD code XL). Also it does say that it can't use all functionality (because my PC isn't AMD as far as I can remember the message said).

superfury wrote:

@reenigne: Ok. One strange thing though: you said that it's bootstrapped using a program in the microcontroller's flash, which isn't accessable by the XT. Then what is that program even doing, if anything at all (sounds like a ROM, which is written to, but unreadable by the CPU, thus making the write to it useless to perform at all)? If the code to be executed isn't accessable by the CPU in any way, how can you even run it?

This bootstrap program is the initial code that is loaded through the keyboard port by the microcontroller (I guess it's not completely inaccessible, since code running on the XT could pretend to be the BIOS and read the bootstrap program the same way, but what I meant is that code running on the XT can't write to the microcontroller's flash).

The bootstrap program hooks some interrupt vectors to provide some services (including bidirectional communication over the keyboard port), then waits to load another program over the keyboard port (the one provided by the XT Server, which may be the sniffer trace program or something else).

reenigne wrote:

That would be great. I've been working on an emulator as well but it's been pretty slow going so you might very well get yours finished first. If so I think there are quite a few existing PC emulators that would like to be able to use that code.

Can't speak for others, but The Borg are watching 😁