Yes, you need at least thermal paste and heatsink. They were usually actively cooled in computers from that time. You can probably get away with a large enough passive heatsink and good case ventilation. I had tried to run mine without a heatsink, it got extremely hot after a minute or so and I had to turn it off.

Heatsink + Fan was common back then on Pentiums. Never saw thermal paste, though. My P133 and a friend's P90 don't have thermal paste, but heatsink + fan and run fine to this day.
Not sure about the MMX, though.

I had tried to run mine without a heatsink, it got extremely hot after a minute or so and I had to turn it off.

Being teenagers back in the 90s, a friend and I also wondered how quickly Pentiums get hot, we turned the PC on with our fingers placed on the CPU, I think after 10 to 15 seconds we had to turn it off. Within seconds, the Pentium was cold as ice again. Oh well, good thing the Athlon XP didn't exist yet 😁

Remember Deskpro from Compaq with P200MMX and cooled by passive aluminium heatsink only, without fan. But that was a big piece of aluminum, with many thin ribs . If you have PPGA CPU, better use thermal grease between heatsink and heatspreader. Ceramic CPU was without grease or sometime with white silicone grease, IIRC.

If you have no air circulation, I would use a heatsink + fan combo. If you have other case fans (e.g. intake fan), then you can probably get away with just a heatsink.

For example, I have a Pentium 200 setup where I rely on an 80mm intake fan to provide circulation.