The instructions for Windows 3.1 worked, thank you! I'm not the biggest fan of the Windows 3.1 interface, but this makes it a lot better.
And your tip gave me a good hint, I googled for "Windows 95" and "Save on Exit" and found the solution. This can be done using the Policy Editor which is not installed by default.
Insert the Windows 95 CD, go to "Control Panel > Add/Remove Programs > Windows Setup > Have Disk...", point it to "D:\Admin\Apptools\Poledit" and click OK. It is now available in the Start Menu under "Accessoires > System Tools > System Policy Editor".
Run it, go to "File / Open Registry", choose "Local User" or "Local Machine" depending on what you want, and under "Shell / Access restrictions" you'll find an option "Don't save settings on exit".
In theory, you could set the corresponding registry setting manually without poledit (supposedly HKCU/HKLM / Software / Microsoft / Windows / CurrentVersion / Policies / Explorer / DWORD NoSaveSettings=1). This didn't work for me though.