Because the clock uses a crystal, and crystals resonate based on their physical dimensions/properties, it won't slow down with low voltage. With low enough voltage, there is a point where a crystal stops resonating properly (too weak, just stops, or oscillates weird), but that causes the time to be off wildly.
So a low voltage battery shouldn't slow down your clock and a high voltage battery shouldn't speed it up.