ADB is not hot swappable, in theory.   Apple warns (warned) that it could cause logic board damage.   Specifically, causing a change in voltage results in a current which is proportional to the rate of change of the voltage, or vice versa.  So if you rip the plug out fast, you may cause a current (or voltage) spike which will damage ADB circuitry.

In practice, it is rarely a problem, but it can be.

Also, after being swapped, the host system will often only operate the mouse at the lowest cursor speed.  There are ADB Reset utilities which can be run after swapping to get the Control Panel settings to apply properly.

Presumably, KVM switches contain bypass capacitors which will moderate peak currents caused by rapid disconnection/connection.