An effective tool to remove small amounts of stuff (like dirt, lacquer, marking, glue) from damageable surfaces is an India rubber.

Some shops sell sets of erasers for exactly this purpose in different flavours (i.e. different in hardness and mixture of the compound). See it as a kind of extra soft grinder. You might do some experiments on the inside of the case before touching the visible surface. For example, try to apply different pressure while using the rubber, as the amount of heat produced by friction affects the result. You also can adjust the degree of matt or glossy finish by choice and usage of the rubber.

I prefer soft erasers without extra colour pigments.