Depending on how rough the surface is, you may have success with some valve lapping compound from an auto parts store. There are several grades of coarseness, so perhaps one of those would work for you. Or you could work your way from coarse to fine, as when grinding any other optical component.