Back on the new/old OS idea: Usual disclaimers apply, I am not a coder, pie in the sky, etc. But how about this for older 68ks?
Take MachTEN's design approach. Make the "new" OS a virtual machine: a process that seizes the CPU from the co-operatively multitasking Mac OS/System software, and runs pre-emptively and with memory protection within itself. All screen drawing and other peripheral device handling is dumped back to the OS/Quickdraw, saving having to write new drivers for everything, and allowing native video acceleration, etc. Once the peripheral task is completed, the CPU is handed back to the VM. So System 6 (for example) acts as the "kernel".
eCos would be a possible candidate, whether using the above approach or running natively.
(Yes, I have brought this up elsewhere. Pardon the cross-post; I couldn't find this thread at the time). For starters, there's already a 68k port, albeit for later embedded 68ks, and it is currently developed, supported, and deployed. Given the memory requirements, perhaps (disclaimer above) it could even run on a 128k.
http://en.wikipedia.org/wiki/ECos
eCos (embedded configurable operating system) is an open source, royalty-free, real-time operating system intended for embedded systems / It is designed to be customizable / programmed in the C programming language, and has compatibility layers and APIs for POSIX and µITRON./
eCos was designed for devices with memory size in the tens to hundreds of kilobytes / It can be used on hardware with too little RAM to support embedded Linux /
eCos runs on a wide variety of hardware platforms, including / Motorola 68000 / PowerPC /
eCos Porting Guide
The only
M68k port I can find is for the Coldfire. As I understand it from others' comments here, the Coldfire ISA is a
subset of that for 680x0s, so my very rough guess is that a backport would be feasible. To run natively, I guess there would be a
lot of custom coding involved to get peripheral device support (ethernet, serial, video, SCSI etc)