Have you checked this site?
http://gona.mactar.hu/ATI_Mac/
I have a B&W G3 (also 8.6) with an original Radeon card. For whatever reason the latest drivers give me a slightly wrong resolution that is 8 pixels wider than the monitor's native resolution. But the original Radeon Mac retail drivers from that site work great. I am using 1440x900 over DVI. But I believe as long as your monitor has proper DDC and your card can read it you should be able to do that with analog VGA as well.
http://gona.mactar.hu/ATI_Mac/
I have a B&W G3 (also 8.6) with an original Radeon card. For whatever reason the latest drivers give me a slightly wrong resolution that is 8 pixels wider than the monitor's native resolution. But the original Radeon Mac retail drivers from that site work great. I am using 1440x900 over DVI. But I believe as long as your monitor has proper DDC and your card can read it you should be able to do that with analog VGA as well.