Let's not forget that this isn't some remote attack that can be run as soon as the machine is online, the code has to actually run locally on the CPU before it stands a chance of reading protected memory. Everyone is worried about web browsers and the vulnerabilities being exploitable through JavaScript, but only recent JS interpreters support what is required for this, and who runs a modern web browser on their retro PC anyway? Just use an old browser, turn off JavaScript, or use a patched web browser - Firefox reduced the precision of their timers recently to make the attack impractical.
It hardly makes the CPUs "unsafe". In fact if you go back in time looking at CPUs, pretty quickly you reach a point where they are too slow to run a web browser with accurate enough timing to execute the attacks anyway.
Really, this is no different to any other malicious code, be it a virus, trojan, etc. If a virus exploits a bug and gets into kernel mode then it can read everything as well, and a lot faster than Meltdown can. Most people try not to run viruses on their CPUs, and Meltdown/Spectre code is no different.
The real issue is people who need to run untrusted code as part of their business - namely cloud providers like AWS. They are the ones who really need to fix the problem or their core business model goes out the window. For everyone else, it's nowhere near as severe.