You'd still not totally eliminate the chance of a jump as the switch sat between 2 contacts I don't think. The I've had nasty arcing noises out of video switches when the switch jammed between 2 poles before now - a bit unnerving. Remember ADB has a voltage running down 2 wires...
For video, hardware switches also tend to screw up your video signal as they aren't usually very well shielded and all the signal wires are split up and fed around the box individually (at least on the cheap ones I have).