This is kind of tricky because there's a lot of variables...
First off, gaining DOS experience, I don't know how experienced you are with DOS, I just know that you are interested in running actual hardware that runs DOS. Overall, working deductively, that would put - based on your software choices - an 80486 DX2 a minimum, and a Pentium III would be the highesst I'd go. Ideally it sounds like a Pentium II would be the best solution IMHO because it's right at that magical point where the old-world IBM compatible elements still existed (dual floppies, FM on a sound card, older, weird, graphics modes), but the new elements were coming in (3D AGP accelerated graphics, memory sizes above 256MB, the ability to run a newer version of Windows like 98 or Me really fast).
As one who has tried to make an all-in-one system, you will most likely run into more challenges than people realize. Pentium III was when they started to phase out the ISA slots for PCI and AGP slots mostly. I had a Pentium III that acted as an "all-in-one" solution, but I was running DOSbox on Windows 2000 SP4 and Nuvie, and Exult, and all these other modernized custom engines to get the best experience - while still playing my modern games on it (this was in 2003-2010) - so you're still running Emulation. Might have a better experience on a slower Pentium III (ie below 667MHz). That system was retro to a T - it was in a mid-80's AT case.
Pentium II is what I'm always told was a good one for retro-stuff because that's the last refuge when options like dual floppy drives and ISA slots were pretty much a given. I could see that being a very good option, though you still might run into trouble.
The Pentium era is a solid choice but things get gradually more expensive as you go back before the Pentium 4 nowadays. The pros to it is that it will run pure dos extremely well, and Windows 9x will also run on it, and you can actually bring it UP to a level close to a Pentium II with a PCI 3D accelerator and gobs of RAM (up to 256MB depending on your motherboard).
The lowest thing anything you want will run on based on your software list is a 486 DX2-40/50/66 model. At that point you might be able to inch somewhat into 9x territory but I don't know what you'd be wanting to get into. Older 9x stuff will run happily on a 486 system but anything past about 1996-1997 or so won't run well or at all.
When it comes to laptops - the Pentium era is when it's at, the ideal is a 640x480 pixel active matrix laptop computer with a SoundBlaster compatible (ESS, IBM MWAVE) sound card installed, and a trackball or touchpad for the pointing device. My top choice has been NEC versas because the prices have not gotten as insane yet as the Toshiba/Compaq/IBM stuff has and parts have been fairly easy to find for a laptop nearing 25-30 years of age.
The thing is, the Pentium came out when 800x600 started to be a thing. I have a Pentium 75 laptop - a NEC Versa P/75, and that laptop is 800x600, so I deal with "letterboxing". but if you can find a P/75 at a reasonable price you could also find an older 486 base unit (versa E/M/Ultralite) with the 640x480 Active Matrix and just latch the screen in place on the Pentium if you want 640x480p (which won't letterbox/leave blank lines around the screen at lower resolutions not divisible by 640x480). That was a big reason I chose the Versa ecosystem right up to a docking station. That said, you might prefer to only own one machine it seems. Which makes it tough. If you're willing to spend, a IBM ThinkPad 755CD or Compaq LTE Pentium 75 model might be the ticket as they came in 640x480 and came with a Pentium or 486 CPU and SoundBlaster compatible audio.
Thin Clients and by extention embedded stuff can be tricky and you have to know exactly what you are getting in order to have one that won't be a fiddly project you'll wind up wanting to throw out the window. There's a lot of small stuff like the fitPC and MediaGX based stuff that you have to be careful of that might not be able to have FM sound, or might be too fast for your application (crash/stack overflow). also, not all thinclients are x86 compatible.
As for desktop machines of the actual era, they can be pieced together over time for relatively cheap by social networking and/or just hunting around the internet at places like E-bay, Vintage Computer Federation forums marketplace, and Amibay. You could even luck out and find another old system somewhere, while they are scarce, they still exist.
Just know, most likely, if you get into hardware, you probably will end up getting more systems anyway. That's probably the biggest risk of all (but the most fun one). I've owned over 60 machines in the last 20 years doing this kind of stuff (I started before it was a "thing") and one thing I've found is once the hardware bug bites, it bites hard, and next thing you know you're buying $60 shelving for your closet to put your machines up neatly when not being used. 🤣.