what you are looking for are stereo line mixers
Behringer RX1602 can be had pretty cheap usually, I got mine for less than the super cheaptastic 8ch audio switchers cost.
there are a few others but those are usually more expensive, they're all single rack units, so decently "small"
Alesis MultiMix 8 Line
ART MX822
Ashly LX-308B
Rane SM82S
Tascam LM8ST
etc. there's plenty more
I recently got a Mackie LM3204 which is basically the holy grail with 16 full stereo channels, but it takes up 5 rack units, still managable but a lot larger.
for our purposes you really don't need anything fancy imo so I'd keep a lookout for the behringer.
the mackie has some neat features that are very useful for the purpose of mixing together midi modules and the like because it has "solo" switches and a seperate mix channel.
the behringer allows you to mute single channels, so you can still turn single channels on and off easily, not all of them feature that possibility, so I'd stick with ones that do so you don't have to change the mix constantly if you want to turn a channel on or off.