well, thats because crystal oscillators drift as well. Thats why rock stable oscillators require a crystal-oven based oscillator, and the most accurate oscillator is a temperature controlled laser pumped rubidium crystal based setup, like what is on board a satellite.

This is why the advent of PLL has came along, to account for this in radio circuits, etc... But regular crystals jitter and drift during operation, and regular cheap old crystals are only accurate to roughly 100ppm.