I've also hot swapped ADB on numerous occasions without problems. From comments above, I gather that I need to at least ensure that whatever switch I use is a break-before-make type.