I tried a CF for DOS once, and started getting write errors on the 3rd time I reinstalled a game that I was having trouble with. Nothing else had been done with it. I suppose that was just bad luck, but it did reinforce my lack of trust in the number of rewrite cycles Flash manufacturers claim their chips to be good for. That card quoted 100K cycles - 100,000 is a long way from 3. I never found Flash cards on game consoles very impressive either, they fail frequently. Most of my SD cards for my camera have been okay, but they've probably only been overwritten about 20 times so that doesn't mean much.
If I try the CF thing again, I'm going to break it into partitions so if 1 partition fails I can easily move to the next. But really, I'd be more inclined to go shopping for old SCSI drives. I haven't done that anytime recently though, so maybe they're all dying off at this point.
I can't say I've tried using a larger(younger) IDE drive on a 486, but I think you could just use a manufacturer utility to limit their size to something the 486 board (or a suitable, slightly newer controller card) will tolerate. I don't know if size limiting is enough, or if that leaves any other compatibility problems.