They'd launch as a program, then sieze control of the CPU (
unco-operative multitasking
), and make Mac Toolbox / System calls as required for display, IO, etc.
MachTen's secret sauce was the VM that their flavor of UNIX ran inside and said VM was undoubtedly pretty nasty to write. (And required a heavily hacked version of BSD to go on top of it.) It undoubtedly wouldn't be *impossible* to recreate something like it so you could use something like µcLinux as the hosted OS but it's going to involve a lot of antique Mac programming expertise. (Remember, you don't have a MMU to work with so you can't "according to Hoyle" virtualize on the 68000. If the guest OS is hardwired to, for instance, want to use the low memory global space for itself (like the Mac OS wants to) it'll have to be rewritten to respect whatever boundaries are necessary to preserve a working System in memory to make calls to.)
Probably the least arduous way for someone who *isn't* a Mac programmer (and doesn't want to learn to be one) to make, say, µcLinux go on a 68000 machine would be to do what most 68k (or Old World PowerPC, for that matter) compatible Linux/BSD distributions did, which is use a "bootloader" which is actually a MacOS program; let the hardware get initialized and the system running the normal Apple way and then run an evil binary which disables interrupts, clobbers all the low memory variables, and sets up the necessary vectors to point at a Linux (or other) kernel which has sufficient device driver support to take over loading the rest of the system from a dedicated SCSI hard disk (or partition). I could be wrong about this but I'm *reasonably* sure that all the hardware interrupts in the Mac point to vectors in low memory and not directly to addresses in ROM... *checks "Inside Macintosh... aha*:
The MC68000 recognizes seven different levels of interrupt, each with its own interrupt handler.
The addresses of the various handlers, called interrupt vectors, are kept in a vector table in
low memory. Each level of interrupt has its own vector located in the vector table. When an
interrupt occurs, the processor fetches the proper vector from the table, uses it to locate the
interrupt handler for that level of interrupt, and jumps to the handler. On completion, the handler
restores the internal status of the processor from the stack and resumes normal execution from the
point of suspension.
So you should be able to completely hijack a Mac Plus with a pretty trivial binary and said takeover should hold until a hardware reset occurs. (At which point the memory mapping hardware that movies the ROM down to $000000, moves the RAM up, and resets the CPU to start executing at zero, thereby undoing all your hard work.) The downside is that you lose the drivers in the System/ToolBox ROM but on the Plus side for a µcLinux geek writing a drivers for the parts you really care about (serial ports, SCSI port, and Display/Mouse/Keyboard in about that order) may well be easier than trying to learn all that MacOS arcane-ity. (Remember, you let the ToolBox ROM intialize everything so with any luck you should be able to take everything over in a known state.) As noted, really all you get by making a custom ROM is the ability to skip having MacOS on the machine at all. (In *principle* you could construct a bootloader which mimics as a working System installation and does the hijacking for you but System and the ROM's boot procedure are so tightly intertwined that's basically Mission Impossible.) I guess if you're interested in *only* the Mac Plus hardware and never want to touch System/Toolbox *ever* making the ROM makes sense, you need to write full devices drivers anyway, but, well... seems almost like you might as well at least verify your drivers by doing the "evil binary" thing outlined above and running some test programs linked to them, if not the full OS kernel, before burning said ROMs. (Otherwise you'll have to either do a lot of frustrating blind testing or develop your ROM in as hardware-accurate-an-emulator as possible, which pretty much points to MESS.)