Ah, nice detective work! What are you using as an assembler? I've been using GNU as, which does have some annoying syntax, like the % before register names. The various addressing modes that GNU tools expect look weird, too (it's called AT&T syntax, I think). That's probably where the confusion comes from. If I had something that used the more common syntax, I'd use it instead.

As for the sound distortion, hmm. I noticed my sound code doesn't touch the ASC register at $802, which the MAME/MESS source code says is a control register. Maybe messing with the contents of that register would help...