You haven't specified the type/class of the computer in question, neither the operating system.

In DOS: in general - apart from the evident inconvenience factor or being a dingle-dongle external gadget - I think the common drawback is that you need to load bigger TSRs for them. External SCSI devices may be an exception, regarding the footprint in memory. USB devices are better to use in a native environment as Windows, Linux etc.
I have been using parallel port devices based on the Shuttle chip. External HDDs and DVD-ROM. The DVD-ROM is tricky, because only certain types work (those that confirm to the earlier ATAPI standard). These are slow but work reliably with small form factor DOS rigs.
I also have a few external Jazz and LS120 drives but these are more suited to Pentium class machines.

I've never had any drawbacks to a USB CD/DVD drive within a native environment (performance, reliability, etc) - that's how I generally prefer my machines setup in fact, because I can have the disc drive up on the desk by the keyboard/monitor, and stuff the tower away wherever is convenient. Some USB external drives also have built-in USB hubs, which can make wiring much simpler if your other peripherals are USB. Some will require all power to be drawn from the computer, usually with multiple USB connections; others just bring along a brick.

In my experience USB floppy drives are divided into the "expensive and decent" and "cheap and horrible" (usually only reads diskettes it has written) categories. I think TEAC made one of the better USB-based external diskette drives, if I'm remembering right.

As far as "other choices" - there might be some FireWire external optical drives out there, but I'd expect them to be very expensive if they exist. And it of course requires the computer to support that. There's also SCSI that will at least give you CD-ROM drives, but they're usually older and slower models if you can find them.

Finally, depending on what kind of wiring you have available, you could always rig up the cables to just have the CD/diskette/etc sit "external" but be plugged into the motherboard via normal channels. Of course they'd have to be fairly close to the case, but depending on the exact situation, it might be workable.

It would be a Windows 98SE system primarily aimed at DOS games. Either 440FX or 440BX based.

It seems to be possible to find an external SCSI CD drive enclosure that pretty much solves the CD/DVD drive problem; but I am unsure about floppy. Should I leave it as it is and only plug in in when needed (hopefully in rare cases); or screw it inside the chassis if there is such a possibility.

I vote for the first option. With that system, you can even manage floppy images in windows, use a virtual floppy drive etc if need arises.
(Of course, there is no harm in mounting it internally if you know for sure you wont need the place it occupies in the chassis for other devices.)

Yeah. But what about CD audio for DOS games? If I were able to find some internal-to-external SCSI adapter (like MediaVision one owned by Cloudschatze), that would pretty much solve it.