Mmh no it's not as easy as I described. Tried again now, plugged a USB key, I got the "found new hardware" popup -> detected a new "USB mass storage" that then disappeared, and now while the computer is not frozen, explorer kinda is with the start menu being unresponsive. I didn't get the "removable hardware" icon in the tray, and in device manager I simply have a new "USB root hub". Something's not right with that USB config.

Try using nusb 3.3 instead of 3.6, it was a bit more stable in my experience.

That said, I completely gave up on USB storage under Win98. A while ago, I was in a similar situation and tried a bunch of things but I couldn't get it to behave reliably. Instead, I now dual-boot Windows 2000 + SP4 alongside Win98 SE and use USB storage exclusively under 2k. No issues whatsoever.

Thanks Joseph_Joestar. This is indeed getting pretty stupid.
I thought I'd try a USB mouse (basic Logitech RX250) instead of USB keys to see if there was a difference. As soon as I plugged it, computer rebooted (no BSOD - straight reboot). Then in Windows the mouse got detected and started working. I removed the mouse, computer didn't freeze. So full of confidence I plugged my first USB key, detected fine, great. Went to unplug it - yay, first time it worked without freezing the computer! Tried another USB key, same thing - detected fine, and was able to remove it smoothly. So I went all crazy - tried to connect two USB keys at the same time...first key OK, and as soon as I connected the second one, computer froze :')

This looks really unreliable indeed. Still I'll follow your suggestion and will try nusb 3.3 to see if I can feel a difference.

For NUSB to run stable, you must follow the installation instructions exactly. If you don't, it simply won't work reliably or not at all.

I did to the best of my knowledge. Rebooted in safe mode with a PS2 keyboard, removed all USB drivers and devices, reboot, let the computer detect it all, reboot. Anything else I missed?

OK I did reinstall nusb3.3 and all is working well, so it seems the 3.6 version was the culprit. Thanks for the help.