It could also be a worn-out switch in your mouse. The switches are not hard to replace and are generally the same as found in PC style mice (or were years ago, not sure what's in USB mice these days). Anyway, I used to pick up cheap three button PC mice so I could steal the switches for Mac mice. Bought separately the switches were upwards of $10, yet, I could get a mouse with three switches for less than that.

The switch is supposed to do something both on the down and the up and it sounds like its doing the up part as soon as you finish the downward motion.

Sorry, I'm so vague, but I don't remember the details. It sounds like the button being held down is not being detected. When you hold the button, the computer thinks the button was simply clicked, and this is a possible failure in the switch, which is the component of the mouse most prone to failure.