What mouse.sys it is, did it come with DOS (which DOS version?) or with some mouse?
What era BIOS it is, XT, AT or PS/2?

I am just trying to figure out if your DOS, mouse driver and BIOS predate the PS/2 era. If they do, then they don't talk to PS/2 hardware.

I'm using effectively an AT system(except CPU), with a 80186 CPU and the Generic Turbo XT BIOS v2.5 loaded. After looking at what happens, the second PIC isn't set up by the BIOS, so the software never receives an IRQ13. The mouse is sending packets, but the interrupt handler never fires because the second PIC isn't configured (all ICWs are zero and the first interrupt controller isn't set up for master/slave configuration).

Is there a AT BIOS out there that will run on a 80186 CPU?

I also think you need to emulate XT system for XT BIOS, or AT system for AT BIOS.

But instead of PS/2 mouse, you could emulate a serial mouse instead, or bus mouse if you have a driver. Microsoft has a mouse driver, but IIRC it does not ship with MS-DOS. Cutemouse could work if you want a real mouse driver. I think it also supports Mouse Systems mode to give you 3 buttons.

Or, you could do what I think DOSBOX does, it already provides mouse interface at int 33h. The mouse movement magically ends up there, without hardware emulation of PS/2 or serial or bus mouse or whatever.

I've implemented a quick and dirty fix: setup the correct icw values on the slave PIC, when the CPU initialises the master (port 20h/21h) to single PIC mode and it's emulating a 80186/8086 (which in my system always uses a XT BIOS, which shouldn't touch the second PIC). The verification of the master-Slave IR lines and base interrupt vector then works using the default hardcoded settings (unless the BIOS overwrites it by programming the second PIC by sending the ICWs to the slave after the master(20/21h) is set up (after the respective master ICW is sent (which sets my default value in the slave), the slave ICW sent overwrites this value if the BIOS supports the second PIC, thus giving compatibility to both XT and AT BIOS. After the ICW 4 is sent/skipped, the emulator forces the master/slave bit in ICW 1 set to support the second PIC.

I notice that the custom driver seems to detect the mouse (it sends a command to start scanning to the mouse. The result bytes are also read from the 8042 controller, as do the packet bytes sent during mouse movement or button presses). The only thing I notice is that after the mouse is set up, the PS/2 keyboard stops working for some reason? (Scanning turned off or mouse input is overruling keyboard input (shouldn't be, since the 8042 prioritizes keyboard over mouse input when loading from the PS/2 mouse&keyboard to it's output buffer (setting the output status flag bit 0 and 5(Depending on mouse or keyboard byte), and raising an IRQ1(keyboard) or IRQ12(mouse) (the IRQ one not in the buffer is lowered). This process is called again when port 60h/64h is read and the output buffer is empty. This is done to support either the same IRQ routine to read more input, or letting it finish, after which the set IRQ1/12 line raises the next interrupt handler to read the next byte buffered (until there's nothing left in the keyboard/mouse buffers.

I'm currently testing using the "DPSMouseDrv PS/2 Mouse driver version 0.9" driver.
The mouse.sys (I believe it's either from the ms-dos 6.22 bootdisk, or installed with Windows 3.0) still does nothing with the 8042 controller. Windows 3.0 is booting in real mode though(I see a mouse cursor during boot, which disappears when the program manager loads. This might have something to do with the driver not supporting graphics mouse cursors?) 😀 I'm using a bootable 1.44MB floppy with MS-DOS 6.22 and mouse drivers, whose autoexec.bat runs win(.com/.exe?) from drive b:, which is a floppy with Windows 3.0 created like this:
1. Install Windows 3.0 setup for real mode (MS-DOS according to setup), with PS/2 mouse and 104 key keyboard settings.
2. Removed most applications from the windows directory and system directory. Also removed 286/386 executables (emm386 etc.) and unused drivers.
3. Copied the hard drive directory (about 2.6MB) to a 2.88MB floppy disk image.
4. Smartdrv.bin, autoexec.bat, mouse.sys and config.sys moved/appended to the dos 6.22 bootdisk.
5. Start up my emulator with the MS-DOS 6.22 bootdisk in a: (Floppy #0 in my emulator) and the windows 3.0 floppy in b:. It takes a minute (during execution of the mouse.sys and mousedrv.exe), but after that Windows 3.0 boots and shows me the Program manager? (the main thing that has the shutdown options and control panel shortcuts etc.), with it's groups. It's unresponsive with the mouse driver loaded though. Without the mouse driver, the keyboard works, although it's too fast. (Pressing alt-f4 shows and closes the terminate windows window in a flash).

It seems the BIOS needs to support PS/2 to use the mouse drivers. Is there an option rom (loaded at c8000-dffff) that adds the required BIOS extensions (like int 15h function c2h)?

How does the microsoft mouse.sys detect the ps/2 mouse? Is this handled by the BIOS? Or does it poll some I/O ports?

If it's handled by the BIOS, is there a BIOS out there that will run on a 80186 processor and supports the PS/2 mouse?

If it's handled by software while booting, how does the software (e.g. microsoft mouse driver) detect the PS/2 I/O port? I don't see any accesses to/from port 0x64 when debugging the driver. The BIOS I'm using doesn't have the INT15h support for PS/2 mice (Generic Turbo XT BIOS v2.5).

Fixed the problem by disabling the PS/2 mouse when running an PC/XT machine, using a serial mouse instead. When emulating an PC/AT, the serial mouse is disabled and the PS/2 mouse is used.