Hi.
I suppose both FDD are on the same cable. Did your cable look like this one (with the twisted part on the ribbon, in red)? It should and the wiring should be done as I indicated.

Yes, it's hooked up just like that. It works fine in DOS. In Windows it accesses the A drive when trying to access the B drive.

Could Windows need in need of a better driver?

Could you try installing windows 98 and see what that does?

I'm not sure a reinstall will work but I can always try. Weird that it accesses only one drive. Hmm.

Experienced some similar weirdness in W98SE when trying to install a second floppy drive a while ago. Both drives worked perfectly in DOS mode.

Fixed it by booting W98SE in safe mode for any device ghosts to display in the device manager, uninstalling all floppy drive and controller entries, and finally reboot to normal mode letting Windows reinstall the controller/drives.

Seems W9x doesn't like adding new floppy drives after OS installation.

Yes, it's probably a registry issue from a bad configuration that occurred somewhere. DOS is using BIOS calls & interrupts with the CPU in real mode to command the floppy controller. The Windows 9x uses their own way to talk to peripherals. Through different software layers from the user interaction on the GUI to the kernel driver passing by different APIs. You can try to see what happens in a MS-DOS Prompt. Because it is a special VM that have a direct access to the hardware but work with an environment exposed by the host. That can be instructive to see the behavior of the FDD there. I think it is important to try to understand the cause of the issue. But if you just want a working PC quickly, the solution of @kaputnik can resolve the issue, it is worth the try.

as long as they are put into the bios correctly they should work unless one is dead. windows or not.