Binary recompilation from 68k to OS X - why not?

[Dennis Nedry]

If this was simple, someone would have done it. This proves to me that there is a lot that I am not aware of concerning what all goes on between 68k apps and Mac OS Classic.

I am wondering why, instead of a complete system emulator approach that boots Mac OS - a computer in a window, what makes it so difficult to recompile 680x0 binaries into Mac OS X binaries? Instructions themselves can be run through a 68k emulator or be directly transcoded. A-Line traps can all be emulated, there is a reasonably finite number of them. These things don't seem to be incredibly difficult or complex.

To me, it begs the question, what else is missing? If you trascode in 68k instructions and replace A-Line traps with function calls, what is left unfinished?

A call to QuickTime or other system extensions seems to complicate things, but what prevents us from recompiling these necessary extensions into the app in a similar fashion?

It seems possible to me that, say, ResEdit or TomeViewer could be recompiled into a Mac OS X native application that uses Mac OS X's functions for drawing windows, buttons, open/save dialogs, etc. What is the huge roadblock that I'm unaware of that prevents this?

 

[Emehr]

The classic Mac OS Toolbox (all of the functions that are Mac OS-specific like drawing to the window, managing files, creating menus, controls, and dialog boxes, etc.) and the Cocoa APIs for Mac OS X are two completely different beasts.

Classic Mac OS applications are deeply interwoven with Toolbox calls and there is no direct one-to-one replacement with Cocoa commands. All of the straight C stuff can be ported with little or no effort, but anytime you want to do anything Mac-specific you have to rethink pretty much everything. Graphics are completely different, for example. In Mac OS, you have to set up your own offscreen "windows" (invisible to the user) so you can make your drawing calls to it then copy it to the screen to avoid flicker. In Mac OS X, double-buffering is handled automatically. You just draw directly to your window.

Not only that, the way Mac OS and Mac OS X handle user and system events is different too. In Mac OS, you have to regularly see what events are in the queue and branch off depending on the event. In Mac OS X, all of that stuff is transparent to the programmer. You basically just have to associate which parts of your interface are associated with what commands.

 

[Dennis Nedry]

Would it be possible to emulate toolboxes on one end for the 68k app and then on the other end provide Mac OS X toolbox hooks that reflect and affect the state of the emulated toolboxes? The overhead seems pretty big, but maybe with a fast Mac it would be tolerable?

Are the toolboxes so massive that any attempt to do this is not humanly possible? We could start with System 6 if that would help.

Or is there a simpler, manual approach - is there a way to break an app back down to its "source code", tweak it yourself to work again, and recompile to a different platform with human intervention?

 

[Trash80toHP_Mini]

A PCI Mac-on-X Card, like the old DOS-on-Mac Cards probably makes more sense. Run the "real" hardware with "real" ROMS and throw the display up in an OSX window. All the I/O would just need to look like SCSI or RS-422. ADB emulation on a Mac-on-X Card could be a problem though, that still needs to be licensed from Apple. Serial works just fine for KBDs, Mice, Trackballs, Digitizer Pads etc.

Dunno, hardware is always easier than software . . . especially so with existing firmware and software libraries . . . at least in my experience.

A 68k/PPC ThunderBolt MacBoX™ might be an interesting toy, now that I think about it. [}] ]'>

 

[Gorgonops]

Re-compiling static binaries for one CPU architecture into binaries for another is a rarely-used emulation strategy for a number of reasons. Issues about API translation aside, machine code targeting a CPU directly simply isn't particularly analogous to most compiled languages; CPUs are complicated state machines with their own unique particulars regarding register layout, instruction behavior, quirks, etc, so it's usually necessary to emulate the behavior of said state machine to some degree or another. Even emulators that employ dynamic recompilation techniques generally don't produce code which constitutes a full-up "translated binary". They usually resort to translating and caching snippets into a sort of psuedo-code that is still ultimately executed via a "CPU engine". The translation step "just" speeds things up by eliminating the need to repeatedly perform steps like instruction decoding. There have been some OSes requiring emulation as a core service that have incorporated the ability to cache and save "blobs" of translated code for future use... off the top of my head I'm pretty sure that several IBM mainframe OSes do it, but it's still not strictly a "recompiled binary".

It is of course possible to use a disassembler on a binary, produce a version of it composed of assembly language commands, and run that through a "cross-assembler" targeting another CPU, but that rarely works with non-trivial software. Your disassembler has to be smart enough to figure out all the external library calls and translate them sensibly, it has to be able to accurately distinguish what parts of the binary are actually "code" and what parts are "data"... etc. For something like a classic Mac program that is intimately interwoven with the ROM and OS it's just not likely to work for any real-world software.

That all aside, if what you want is an API translator that works with a classic Mac binary and makes it "native", in the sense that it runs without directly requiring the CPU emulator to execute a complete copy of the MacOS such a thing already exists. ARDI Executor is/was a system for emulating a significant subset of the Mac Toolbox and OS APIs to a sufficient degree to run at least some Classic Mac software on Windows or Linux. (It's basically analogous to how WINE for UNIX systems allows at least some Windows programs to successfully appear to run as native X11 applications.) The source code for it is available, so in principle at least you could port either the Windows API or X11 version of it to run on Carbon/Cocoa instead. Executor displays Mac programs running on it on their own "Desktop", but I imagine with some work you could figure out how to make the emulated programs run "seamlessly", IE, on a transparent desktop intermixed with other windows. If you were willing to put in the work to make it happen then you could then take the Mac application you want to run and package it in an application bundle with your modified version of Executor, configured so double-clicking on the icon would start an Executor session dedicated to running the emulated program.

(I've seen WINE for OS X packaged into an App bundle to run a single dedicated Windows .EXE, so it's not a unique idea.)

The code is there if you really want to hack on it.

 

[ClassicHasClass]

Would it be possible to emulate toolboxes on one end for the 68k app and then on the other end provide Mac OS X toolbox hooks that reflect and affect the state of the emulated toolboxes?

Somebody did do this for MacDraw. http://www.dejamac.com/software/macdraw/

 

[Gorgonops]

Somebody did do this for MacDraw.

It's a shame the GreyBox project behind it appears more than a little bit "abandoned".

According to a note on ARDI website the author has already submitted some minimal patches to make Executor compile on OS X. Time to start merging the two projects!

 

[ClassicHasClass]

I think it was too much for Leif at the time, and the Rosetta requirement is obviously a no go now.

 

[bbraun]

FWIW, I've been playing with GrayBox, enough to fix some bugs and extend it enough to get FDisasm working with it.

GrayBox runs a full MacOS, booting with a Plus ROM, and (in the case of MacDraw above) a System 6.0.7 System file, with the desired executable replacing Finder, all of which live inside the .app's bundle.

It then passes through select traps to Carbon nearly verbatim, just doing BlockMoves of the trap arguments from within the minivmac instance into the native ones defined by Carbon headers (with some swizzling of select pointers between the minivmac instance and the host). It is limited to 32bit only both due to Carbon use and doing BlockMoves which include pointers. It's limited to PPC due to no endian swaps.

It doesn't handle the vast majority of traps, allowing them to be handled by the booted system and ROM running inside minivmac. As a result, it gets a bit hairy when trying to add support for new traps, since some are using System 6/Plus ROM, and some are using the host system's Carbon implementations, and they're sharing the same data structures.

It's a neat project, and it is useful for some specific things more or less as is. It runs MacDraw and FDisasm! But the work required to make it generally useful is both daunting, and looks a lot like moving in the direction of Executor. At which point, it may just be better to go with Executor anyway. I'm planning on playing with Executor next.

Unfortunately, due to the requirements of including the ROM and System file, most any work on GrayBox quickly becomes an inappropriate topic for this forum.

 

[Dennis Nedry]

This is an awesome thread, I have learned a lot here - I wasn't aware that any of this stuff was going on.

Unfortunately, due to the requirements of including the ROM and System file, most any work on GrayBox quickly becomes an inappropriate topic for this forum.

Could one simply require that the ROM and System be supplied by the user? These items could be placed at Library/Application Support/GrayBox/ and accessed by any and all software builds without including them in the build.

If GrayBox is written dependent on taking chunks of ROM/System and putting it into the actual code before building it, these code chunks could be separated into a library. This library could at first be shared in the most secretive fashion among developers, but eventually you could develop a way to automatically produce the library from the ROM and System which comes full circle to requiring that these items be provided by the users out there and NOT provided by the developers.

 

[bbraun]

Well, there's lots that could be done, but discussing how to circumvent the rules is already into uncomfortable territory for this forum IMO.

But, assuming that was resolved, you'd still be left with a 32bit PPC-only solution that still requires quite a bit of work to be generally useful, and once all that work is done, it sure seems to me like it'd ultimately end up looking a lot like a rootless Executor.

FWIW, following the instructions in the Executor README, it built and runs fine i386 on 10.6 for me.

I hadn't intended to compare performance, but I couldn't help but notice the difference when I ran my GrayBox version of FDisasm against my IIsi ROM annotations vs. Executor with the same thing.

GrayBox:

real	0m23.311s
user	0m19.823s
sys	0m0.140s

Executor:

real	0m1.890s
user	0m0.545s
sys	0m0.316s

 

[olePigeon]

Something like this?

 

[Gorgonops]

(Frying egg sound effect.) "... This is your NeXTstation on Daydream. Any questions?"

They used to sell Mac ROM boxes for Amigas and Atari STs that accomplished essentially the same thing. Which is pretty much the opposite of what the OP wants. ;^)

 

[jongleur]

I tripped across http://www.metamage.com/code/v68k/ looking for something else. Is it useful?

 

[ClassicHasClass]

That's Josh's stuff. He does some interesting things. I don't know if he's around here though.