FWIW, there were ROMless 040 accelerators for 030 and earlier machines which predated the 040 and did not have 040 instructions in ROM. The Presto LC for the original LC for instance, has no ROM on it (AFAICT). The Daystar 040 has ROM and is doing a bunch of ROM patching, but some of these other accelerators are significantly simpler.
I haven't done a whole lot of investigation (accelerators aren't my thing, the difference between fast and slow 68k macs is lost in 20+yrs of advancement to me). The primary areas of concern and/or interest would be around cache: the 040 introduced separate instruction and data caches as well as copyback, and the MMU: the translation control register was 32bits in 68851 and 030, and 16bits in 040.
Of these, the MMU is the biggest concern.
For the FPU, all the ROMs included the SANE FPU emulation anyway, so I'm suspecting any unimplemented FPU instructions would trigger an unimplemented instruction trap and fall back to the SANE emulation.
For the cache, I wouldn't be surprised if the vendors wired in the cache inhibit line to always be asserted, essentially running the 040 cacheless. This would solve the incompatibility problems, and the 040 would still be as fast or faster than the 030. Or possibly running cacheless until some driver software is loaded or some such.
That leaves the MMU. My guess is while not being totally compatible, it's good enough for the relatively limited use of MacOS.
And most of the rest of the differences aren't as much compatibility as much as lack of optimization, like the faster BlockMove for 040. Again, potentially solveable with driver software loaded at boot time.
Anyway, that's just the CPU compatibility, and for accelerator manufacturers, the burden of compatibility was on them. Compatibility between machines with different memory controllers, video, etc. is something entirely different. Not to mention different ROMs with different memory managers. The differences in the memory manager between newer ROMs and System6 seems to be the source of the instability on my LCIII once it got booting.