Interesting timing. I did some experiments on SE/30 screen resolution a year or so back, but I'm just about to revisit the topic:
https://68kmla.org/forums/index.php?/topic/26313-compact-mac-retina-display/
The limiting factor is the horizontal scan rate, which is 22kHz on all the compacts and is very sensitive to even slight changes. As I found in those experiments, increase the scan rate by more than a few percent and all the flyback secondary voltages start to drop significantly.
In principle the flyback circuitry could be changed to run at higher frequency, but what I found is that the internal parasitic capacitance of the flyback transformer itself means that the horizontal retrace period basically can't run any faster than it already does. In other words, with the flyback transformer that's on the SE/30 and earlier Macs, there's just no way to run the whole circuit fast enough to reach VGA resolution.
What I tried in the thread above was to build a second partial analog board circuit: one to run just the flyback with a dummy load in place of the yoke, another to drive the yoke without a flyback. It does work, after a fashion, but the video quality is limited by interference patterns between the two frequencies. That was about as far as I got at the time.
But this week I got my hands on a late 1991 Classic. The later Classics and the Classic II have a totally different analog board to all the previous compacts, including a different flyback model. I'm going to see if this one is any more flexible than the earlier model at running at higher frequencies. But first I need to do a full recap, because the power supply and logic boards are both in terrible shape!
As for the grayscale part, the circuit for the Xceed CRT board is out there somewhere, and could presumably work in this project too. It's the resolution that's the harder part.