Well, the PSP works very well for me, and you can always run CFW on it without actually flashing it, making it permanent or modifying it in any other way. Only downside to that would be that you have to launch it every time you power it on, which can get a bit annoying. However, making it permanent can be extremely quick and easy as well, and undone anytime you want. Plus, some homebrew apps, including emulators, are already signed, so you don't even need any CFW to begin with.
A 3DS, especially the newer model would also be powerful enough, but that would require some actual hacking, and a DS/DSi would be too underpowered, plus, the screens have lower resolutions than the actual systems you would want to emulate, so no go there.
The GPD WIN Max seems like a good option, but it might be a bit too pricy and overkill (I sure would love to have one, but that's besides the point 😁).
The Pandora handheld (anyone remember those?) was also something I really wanted as a teen, but the Dragonbox Pyra (its successor by the same people who made the Pandora) is even better, but quite expensive.
All of those options have removable storage, analog sticks, removable batteries (well, most of them do) and powerful enough hardware to properly emulate PS1 and anything before that. I don't know anything without analog sticks that's good personally. Then again, of course I don't know everything 😉