For joysticks and keyboards, I've had extremely good luck with Goodwill stores. You just have to be patient, check frequently, and know how to disassemble and clean input peripherals. Especially keyboards. There are some really dirty bastards out there. My personal method involves a few days of soaking in automatic dish soap, then regular dish soap, and finally a key-by-key disinfection using Clorox wipes with bleach. By the time I'm done, they're back to a germ-free almost-new condition. I only go through this much effort for really special items: IBM model M or KB-8923, MS Natural, or other front-tilt ergo keyboards that are hard-to-find nowadays. Joysticks and mice are much easier to clean and usually just require a vigorous scrub with Clorox wipes.
I also find lots of good old IDE/PATA CDROM and DVD drives at Goodwill. You just have to know what you're looking for. Pretty much all Pioneer, Plextor, Lite-On, NEC, and Sony drives are good. I'm not a real big fan of Toshiba and Samsung ODD's, so I avoid those.
My only good find on craigslist came this past November when I scored a Dell Dimension 4100 for $10. Sadly, after I got it home and opened it up, I found that it was very used and truly worth only $10. There was so much dust and fuzz inside the case... it was really the worst I've ever seen. After cleaning it out and putting a few drops of oil in the CPU/exhaust fan, it is working OK. Seriously, this wasn't some old P3 system that was used in the early 2000's and then stuffed in a closet. It had clearly been used consistently for the past dozen or so years. Well, it still works and I'm in the process of putting it together as a spare Win98SE box. Even after upgrading it to a 1ghz Coppermine and popping in a GF4 Ti4800SE and NEC USB 2.0 PCI card, I've still only got about ~$35 invested in it. Not bad...
I've also scored a few good CRT monitors on craigslist. Check the free section, because CRT monitors are pretty much worthless nowadays and folks just give 'em away. Seriously, never pay more than $5 for a CRT. Even then, for $5 it had better be in damn good shape.
Pretty much everything else I get from eBay, which I hate. I would shop elsewhere for retro parts if I could.
For cases, I don't even bother with period-correct parts. I just buy new stuff that will fit the old gear.