Obviously the FIFO being circular or not is irrelevant.

For example "REP MOVSB", I don't think there's any other way. The REP prefix and MOVSB opcode are fetched from CS:IP to FIFO just once, which again means they are popped off the FIFO just once.

The string instructions are executed until it is complete, either once, or as many times needed if the REP prefix is in effect.

So that means that after a long instruction, the REP MOVSB is executed without any changes(even without FIFO) and the FIFO always contains 0-6/4 bytes of new instruction data(the 4/6 bytes following REP MOVSB,depending on the time REP MOVSB takes) before REP MOVSB overwrites them? Thus my emulation&FIFO reset is incorrect?

As Jepael says, they are fetched only once. In the meantime, the prefetch buffer is refilled during idle data bus cycles, so by the time the instruction completes, the buffer is usually full again.
This means that if you have code that uses rep movsb to modify data under the prefetch buffer, you should technically prefetch during the rep movsb, not after it.
However, when exactly you should prefetch, and how much you should prefetch, that depends on the number of idle bus cycles. So the only way to get it 100% correct is to completely emulate the data bus in a cycle-exact way.

These should be corner cases though. In 99.999999% of all cases, having the prefetch buffer full at every instruction should do the job. I suppose in the case of a rep movsb, you should prefetch after the rep and after the first movsb (you could do it after every movsb, because it will be full after the first one anyway, might make things simpler).

https://bitbucket.org/superfury/x86emu/src/45 … cpu.c?at=master

I've changed the prefetch to load at every instruction byte fetch from the buffer. I've also added a bit of logic to read REP instructions only once from prefetch. The error in castlevania (BIOS executing STI, interrupt handler, acnowledge interrupt(port 20h), after which the interrupt retriggers because IF==1. Anyone knows if this is correct behaviour?

Edit: It seems that the main code of castlevania is executing a REP MOVSB followed by a LOOP eventually. So it's moving big blocks(many KB at once every REP MOVSB). But it keeps getting interrupted by the INT8 timer interrupt(IRQ0), which nests itself(because of it's STI) once it tells the PIC it's done. Starting the emulator debugger reveals this.(Timing-based events are disabled(PIT, HDD, Keyboard, Mouse, Adlib) during debugging to prevent infinite IRQs, as the handling of timing would have to process several seconds/minutes(time the CPU is paused by the debugger) all in one CPU execution of an instruction. (Lets say PIT0 is timed at 1MHz with the CPU pauses for 10 seconds. Then the timing would process 10*1000000=10000000 IRQ0 flagsets in one go(processed one by one in a loop.).