It's certainly not something you should with high powered equipment. Line level sources might have enough output impedance to keep the final amplification stage from going full Chernobyl.

Jheronimus, what about finding (or making) an audio switch? Having the outputs float might be okay (I'd have to think about it a bit), but I think everyone here is familiar with boneheaded moments that happen from time to time when we have things set up to run safely if they're used correctly, then in a fit of narcolepsy we go and use things incorrectly...