It is a pain, but I wrote unit tests for the instructions (not 100% code coverage yet) and as part of that unit test, if it is run on msvc in a 32-bit build, I also verify that it works the same way on hardware. So for a standard ADD test I would call an macro that looks like this where op, a, c are passed in then I verify the result and flags. I would strongly recommend including actual hardware tests in your unit tests if possible. In this function I have an array of data then it will loop through testing different things

1                __asm {                             \
2                __asm mov ebx, data                  \
3                __asm mov eax, [ebx].var1            \
4                __asm mov ecx, [ebx].var2            \
5                __asm mov edx, flags                 \
6                                                \
7                __asm push edx                       \
8                __asm popf                           \
9                                                \
10                __asm op a, c                        \
11                __asm mov result, eax                \
12                                                \
13                __asm pushf                          \
14                __asm pop edx                        \
15                __asm mov flags, edx                 \
16            }

example data for 32-bit add

1param1 is value 1
2param2 is value 2
3param3 is result
4param4 is starting flags
5param5 is sets CF results
6param6 is sets OF results
7static struct Data addd[] = {
8        allocData(100000, 200200, 300200, 0, false, false),
9        allocData(0xFFFFFFFF, 1, 0, 0, true, false),
10        allocData(1, 0xFFFFFFFF, 0, 0, true, false),
11        allocData(0, 0, 0, 0, false, false),
12        allocData(0xFFFFFFFF, 0xFFFFFFFF, 0xFFFFFFFE, 0, true, false),
13        allocData(0x40000000, 0x40000000, 0x80000000, 0, false, true), // overflow indicates that the sign changed
14        endData()
15};

Well, according to test386.asm's undefined flag tests, it's supposed to set both CF and OF accordingly. Otherwise, it crashes.

I was able to get Win 95/98/2k/XP to boot in jDosbox and I did not set OF for BT/BTS/BTR/BTC

For boxedwine I have some hardware tests for those, but I didn't set or check for OF. Just for fun I set the starting condition to have OF on and off, and using BTS with my 4 sets of data did not result in OF changing. This was on a core i7.

Well, according to test386.asm it does affect the Overflow flag on a 80386sx(during it's undocumented flags POST E0 check). Said check will fail on e.g. Bochs and apparently succeed on a 80386.

Edit: test386.asm source code confirms that(apparently verified against a real 80386sx):

bt386FlagsTest: ; BT, BTC, BTR, BTS ; undefined flags: ; OF: same as RCR with CF=0 […]
Show full quote

bt386FlagsTest:
; BT, BTC, BTR, BTS
; undefined flags:
; OF: same as RCR with CF=0

If it is undefined I guess its not surprising it could behave differently on different chips. That is too bad that there is software that depends on undefined behavior.

There are no undefined flags, only undocumented ones.
Yes, OF is set up to the Pentium 2 according to some rules.
Pentium 2 and later don't touch it.
It's set according to the result of ROL of 32 - bit_to_test.

peterferrie wrote:

There are no undefined flags, only undocumented ones. Yes, OF is set up to the Pentium 2 according to some rules. Pentium 2 and […]
Show full quote

There are no undefined flags, only undocumented ones.
Yes, OF is set up to the Pentium 2 according to some rules.
Pentium 2 and later don't touch it.
It's set according to the result of ROL of 32 - bit_to_test.

Thank you for clearing that up. It is hard to discover those things without having old hardware to test on.

peterferrie wrote:

There are no undefined flags, only undocumented ones. Yes, OF is set up to the Pentium 2 according to some rules. Pentium 2 and […]
Show full quote

There are no undefined flags, only undocumented ones.
Yes, OF is set up to the Pentium 2 according to some rules.
Pentium 2 and later don't touch it.
It's set according to the result of ROL of 32 - bit_to_test.

You say ROL of 32 - bit to test. But isn't it RCR with CF=0 instead of ROL of 32 - bit to test? Shifting right until shifting out the bit(thus RCR val,(bit&0x(1)F)+1)? That also makes sense with the result. If you try to do it with RCL, won't you get strange results? Or is RCL 32/16-b the same as RCR b?

You might get the proper result with RCR/CF=0 but I never tried it.
However, they're not the same - RCR with CF=0 introduces a zero-bit where there wasn't one before. If you ROL (not RCL) an 8-bit number by 8 bits, then carry receives a copy of bit 0. If you RCR the same with CF=0, you don't get the same result.

What I mean is RCL x vs RCR operandsize-x.

Also, UniPCemu does RCR n+1 with all BT instructions(as a 16/32-bit RCR). The results of OF/CF are set accordingly.