A first test would be to attach a second monitor on the (optional) video mirroring port, disconnect the internal monitor and see if the logic board can drive the external @ 800x600 (or indeed any other resolution). If that fails you might need an LC PDS video card.
Many apologies if I'm wrong with either of these statements, but... a couple points there:
A: Isn't the only "video mirror port" available for the 575 a rare third-party device that hackishly pressed on top of the video chip? and...
B: I may be completely misremembering this, but aren't all the monitor lines part of the big nasty connector that the whole motherboard plugs into in the front of the case? That being the case the only way to "disconnect" the internal monitor would seem to be to cut up the internal wiring harness, or at least fiddle with the edge connector. Even if you were to disconnect the wiring at the Analog Board end, if you can do that cleanly, you'd still need to muck with the sense lines in the harness to let the Mac set other resolutions.
EDIT: BTW, if you Google around you'll find passing references in forum threads to people getting 800x600 out of LC 575 boards plugged into "Mystic" Color Classics when the harness has been modded for VGA. (Remember, "Takky" was the radical harness swap operations with a 5200+ series box.) So it's probably safe to say that going to weird lengths to "prove" that the motherboard is capable is unnecessary.
How would one go about doing this?
I think the short answer is you don't want to try, and if you want a higher-res Mac otherwise like your 575 you'd be much better off buying an LC 475 or Quadra 605 and pairing it with a nice Multisync monitor. A monitor stacked on an LC case takes the same space, basically.
Otherwise, if you want to hack up your 575 your best bet appears to be to see if archive.org still has copies of the Japanese-language websites which fully documented the wiring harnesses in Color Classics, and follow the instructions to enable "VGA mode". (It's unfortunate, but a simple "google" doesn't seem to turn up anything but piles of dead links.) The 575 has the same wiring harness and the sense-line setting for "VGA" will unlock the ability to play with arbitrary video modes on the motherboard side. The only difference is that yours will have different settings to start with, since the mode the CC was locked into was 512x384. (
Wild guess, but I imagine the sense line setting for the LC575 matches that for a "Hi-Res" monitor while the CC matches "RGB 12".) You'll just have to remember that without drastically fiddling with its settings or changing component values the only mode the monitor is *designed* to accept and display properly is 640x480@67hz. If you had the wiring diagram you could possibly cut the correct lines and wire a VGA plug to them to disable or mirror the internal, but... do you really want to do all that?
Eh, go for it if you're up to it.