Probably your port. Although what behavior do you see?

If I use my A2M mouse, it works perfectly.

If I use an M0100 mouse (from a Mac Plus,) the button works, and moving *DOWN* works, but moving up does not, and left/right is *VERY* sporadic (any left/right movement is completely random compared to where I actually move the mouse.)

And if your joystick doesn't work (how would you know if you have no way to test it?) then it's the port for certain.