If it was a RAM issue you'd have a Sad Mac, or at least other crashes. I'm pretty sure that the SE, like the SE/30 but unlike the Plus, has separate video memory that doesn't use main RAM.
Unfortunately, I can't find any schematics of the SE video circuits. The logic board schematics online are...