I have had similar problems and I don't have a clear solution for that situation. You would think PRAM zapping would take care of it, but there are times that it won't. Zapping the PRAM is supposed to reset you to some default resolution, but maybe that resolution isn't supported either.
It could be that your particular video card stores the resolution onboard instead of in PRAM. In this case, the only solution would be to attempt switching monitors or booting without a monitor, then plugging it in after it's booted a ways. Using your VGA with an adapter might do the trick. See if you can temporarily set it to 640x480 fixed resolution mode. Try to think of ways to confuse the video card into resetting the resolution.
Where are the resolution settings stored?
The resolution settings are stored in the system's control panel folder. I'm not sure what OS you're running on your IIfx, but if its System 7, there should be a icon that says "displays." But follow my first suggestion, and try using a different monitor to hookup to your IIfx. Resetting the PRAM won't help because the PRAM has nothing to do with display settings to my knowledge.
Zapping the PRAM is known to reset monitor resolution, so the display settings actually are probably stored in PRAM. In OS 8 or 8.5 or so, these settings must have also been stored in a preferences file and restored during boot if the PRAM was reset.