Thank you all for your help and replies. So today I finally had the time to check it out and here's what happened:
I picked my spare PSU from the leftovers, and ironically it's the exact same model as the PSU in the case - Primer LC-B300-ATX. I opened them both, the PSU in the case has some bulged caps about to start leaking in the future, and the extra PSU had all caps looking fine. So I tried the extra one, remaining suspicious, but it's all I had.
Installed it, powered up the PC with the case open, heard a mild pop and saw some faint and non-smelly magic smoke. Turned everything off and checked it out. The smoke really seemed to come from the right, the front panel side (drives). Removed from the case and opened up each drive, a Panasonic floppy drive, an Asus CD reader and a HP writer. Didn't see any signs of something blown up and none of them smelled. Opened up the extra PSU that I just tested and didn't see or smell anything either, not even the smallest resistors/diodes had any signs. At last checked out the motherboard, which was immaculate.
Proceeded my kamikaze testing style one more time without any drives, only the Mobo, CPU (which has a Gelid fan controller connected) and RAM. Post beep, all normal. Connected the floppy drive just for kicks. No power and obviously no floppy seek. Funny enough, removed it and tried the bare configuration again and the CPU fan didn't spin on startup. Removed the fan controller, same thing. After a try or two, it started normally, post beep again, etc. Reconnected the fan controller, all normal again. So I decided to stop and cut my losses. My beginner's money is on the PSU.
Really wish the pop came from the floppy drive as it's the easiest replacement for me. Do you guys have any idea what could have been?
Just to recap, clarify and help guide future readers of the thread with similar problems: this PC has had the same hardware for years, with no issues. It was up until yesterday that I started having the problems described in the OP, so until further proof I'm inclined to think it's just both PSUs causing me various issues.
The motherboard always powers up and post beeps apart from that weird transition after reconnecting the floppy drive, so I'll be optimistic for now and assume it's fine. I didn't connect the HDD at any moment as I really don't want to risk it, it's a heavily configured build and it really would be a massive pain to reinstall everything...
Being pointless to proceed without a new trustworthy PSU, I'll just leave the PC with the bare minimum, install it when I get one, and go from there, drive by drive, with a new floppy one. My doubt is, what modern PSUs do you recommend for old PCs? I imagine they have molex connectors but how do you connect the floppy drives to those modern PSUs?
As always thank you for all your help and replies