I have a B&W G3 (also 8.6) with an original Radeon card. For whatever reason the latest drivers give me a slightly wrong resolution that is 8 pixels wider than the monitor's native resolution. But the original Radeon Mac retail drivers from that site work great. I am using 1440x900 over DVI. But I believe as long as your monitor has proper DDC and your card can read it you should be able to do that with analog VGA as well.
There's a resedit hack for running a specific PCI video card at arbitrary resolutions which someone posted here - in an 8500 under OS 8.something I think. Search?