One of the best cleaners I've found for this sort of stuff is called Oil Eater. It's designed as an automotive/workshop de-greaser but the stuff works great to clean off finger oils and all manner of other crud. For really stubborn stuff, try using it full strength and warm it up in a microwave. Gloves are recommended, otherwise it dries the hell out of your hands. Supposedly it's non-toxic and biodegradeable too.