It seems to me, and as only an amateur programmer I’m more than willing to admit that I could be wrong, the best solution would be to program a highly standardized interface for the plug-in.
Since DOSBox is designed around running in a modern OS, then I believe all of them (with the possible exception of Linux,) are multitasking environments. So, if you set it up so that the plug-in is actually an external program that runs separately from DOSBox then a lot of the backwards compatibility issues could be avoided there by setting up a standard form of communication. As all plugins would use the same methods to communicate with DOSBox regardless of how old they are. The highly published and non-proprietary ISA expansion port seems ideal to use as a model or template for communicating between DOSBox and the plug-in(s). Then you could set up a variable that limits the number of “ISA” ports that are available to DOSBox. I would suggest setting that number to 6 or 8 (like the number of ISA ports in a physical x86 machine.) Then you could, if you wanted, design it so that the Plug-in could over-ride the DOSBox built in emulations if the user wanted.
For example, someone could write a plug-in that acts like a communications expansion card, replacing the COM and LPT ports that DOSBox currently emulates. Or even a Voodoo 2 card to replace the VGA emulation. This could end up opening up security issues (as the plug-in would actually be running outside of DOSBox.) However, those security issues would be separate from DOSBox itself, and not the responsibility of the DOSBox team (not that it would ever really be in the first place.) The sole responsibility would be on the shoulders of the person(s) that wrote that plug-in and on the end user that decided to use it.
As I mentioned above, though, I don’t know if all of the systems DOSBox is designed to run on are multitasking. Not knowing that much about Linux (or Unix,) I have absolutely no idea if it is or not. I know that Windows and the Mac OS X are both multitasking systems though. The problem would be setting up this sort of plug-in system to work on those other OSes as well.
On thinking of the Voodoo 2 example above, that would probably end up opening a second display window (in addition to the normal DOSBox display window.) Not sure if that would be a good thing or a bad thing. Good, if the Voodoo 2 plug-in had some way of automatically switching between itself and the DOSBox diplay window that spawned it. I just don't know enough about programming to know if that is possible.... Now this is going to bug me all night 🙁 Not enough to put the effort into learning a programming language just to find out, though.
p.s. Egads, I didn't mean to make it sound like ISA expansion card emulation was the only goal. Other plug-ins could be designed I'm sure, they would just use the ISA standard for comunications (only a lot faster than an ISA port.) Such plug-ins could include, a cheat GUI that monitors the registers and user selected memory locations while DOSBox is running. Such a plug-in could search for patterns, such as the code that marks a character's hit points in a game, and then the user could alter it on the fly. I used to have something that did that for a Windows game I used to play, don't remember the game though.