Seems like it could be the IMUL result range checks in instructions.h:

1#define DIMULW(op1,op2,op3,load,save)                  \
2{                                             \
3   Bits res=((Bit16s)op2) * ((Bit16s)op3);               \
4   save(op1,res & 0xffff);                           \
5   FillFlagsNoCFOF();                              \
6   if ((res> -32768)  && (res<32767)) {               \
7      SETFLAGBIT(CF,false);SETFLAGBIT(OF,false);         \
8   } else {                                    \
9      SETFLAGBIT(CF,true);SETFLAGBIT(OF,true);         \
10   }                                          \
11}
12
13#define DIMULD(op1,op2,op3,load,save)                  \
14{                                             \
15   Bit64s res=((Bit64s)((Bit32s)op2))*((Bit64s)((Bit32s)op3));   \
16   save(op1,(Bit32s)res);                           \
17   FillFlagsNoCFOF();                              \
18   if ((res>-((Bit64s)(2147483647)+1)) &&               \
19      (res<(Bit64s)2147483647)) {                     \
20      SETFLAGBIT(CF,false);SETFLAGBIT(OF,false);         \
21   } else {                                    \
22      SETFLAGBIT(CF,true);SETFLAGBIT(OF,true);         \
23   }                                          \
24}

Should the res<32767 and res<2147483647 be less than or equal?... or the constants greater by one, but that could flip the sign bit depending on the data size.

according to the logic of Intel.

According to the logic of the intel docs or the real processors?

FWIW, dynamic core on my system does not set the carry or overflow flag with the given example, but normal and simple core do.

According to the logic of the intel docs or the real processors?

The question is interesting. Of course, when we run the example on a real processor, we have CF = OF = 0. But if we look through the official documentation of Intel 80186, where this instruction appeared first... There are two tables there: a table of the strict instruction descriptions and the table of the instruction codes. We find this instruction in the second table (though they don't describe there the code of the variant when the second operand is a pointer to memory). But they don't describe it in the first table! Did they forget to do this, or maybe they didn't do it on purpose?.. Or I just saw a wrong paper? Does not clear.

But, in any case:

1if ((res> -32768)  && (res<32767))

Of course, there must be

1if ((res>= -32768)  && (res<=32767))

Because in my example we have 32767 as the result, and this is not overflow. And, analogously:

1if ((res>-((Bit64s)(2147483647)+1)) &&               \
2      (res<(Bit64s)2147483647))

We must have there instead:

1if ((res>=-((Bit64s)(2147483647)+1)) &&               \
2      (res<=(Bit64s)2147483647))

Of course, when we run the example on a real processor, we have CF = OF = 0.

Did you test this on a 8086 or 80386? As flags behaviour is by times considerably
different from those to the pentium+ generation.

But most likely this is indeed an off by one mistake.

The tests were run on an Intel Pentium IV and on a clone of Intel 80186c processor.