There's a very good chance that it is a RAMDAC issue. Even if there is enough VRAM to do larger resolutions, and even if the memory is fast enough to output the full resolution at lower color depths, the RAMDAC still needs to be fast enough to spit out enough pixels to cover the whole screen.
Ignoring the blanking intervals, at 800 X 600 @ 60Hz, the RAMDAC is spitting pixels at 28.8 MHz.
At 1280 X 1024 @ 60Hz, the RAMDAC is spitting pixels at 78.6 MHz and it doesn't matter what the bit depth is. The RAMDAC is essentially a three port D to A converter which accepts the digital pixel information, be it 1, 2, 4, 8, 16 or 24 bits of color data and then converts that digital data into analog voltage levels at the frequency rate required for the display output.
Anyway, you might, if there is some way to do so, try hacking a lower screen refresh rate onto the higher resolutions and see if that removes the artifacts. That would be a good test of whether the problem lies in the RAMDAC speed.
Ignoring the blanking intervals, at 800 X 600 @ 60Hz, the RAMDAC is spitting pixels at 28.8 MHz.
At 1280 X 1024 @ 60Hz, the RAMDAC is spitting pixels at 78.6 MHz and it doesn't matter what the bit depth is. The RAMDAC is essentially a three port D to A converter which accepts the digital pixel information, be it 1, 2, 4, 8, 16 or 24 bits of color data and then converts that digital data into analog voltage levels at the frequency rate required for the display output.
Anyway, you might, if there is some way to do so, try hacking a lower screen refresh rate onto the higher resolutions and see if that removes the artifacts. That would be a good test of whether the problem lies in the RAMDAC speed.