ISO sample code to create an HFS filesystem or disk image

[bigmessowires]

I'm looking for sample code to directly create an HFS filesystem / disk image on a modern computer, where the HFS disk contains a single file that's supplied as input to the code. Ideally this should be platform-agnostic code without relying on third-party libraries or OS dependencies. The file is data-only, with no resource fork needed. My goal is to to ease the use of StuffIt archives with the Floppy Emu disk emulator by automatically creating an HFS disk image that contains a .sit file stored on the SD card, since this is a common point of confusion for users.

There's an older utility program called HFS Disk Maker that's nearly exactly what I need. I wrote to the author, but haven't heard anything back yet.

If all else fails, I could crack open Inside Macintosh and start reading the HFS specification and mind-numbing details about b-trees and such, but I'm hoping to avoid that.

 

[dougg3]

I was recently looking into doing some Mac file manipulation from Linux for my reverse engineering efforts, and noticed that Retro68 uses libhfs from hfsutils in order to write disk images. I think this might be right up your alley if you can use GPL code. It can even handle resource fork stuff if you end up needing it.

See ResourceFile's code for saving as Format::diskimage for an example.

 

[eharmon]

I was recently looking into doing some Mac file manipulation from Linux for my reverse engineering efforts, and noticed that Retro68 uses libhfs from hfsutils in order to write disk images. I think this might be right up your alley if you can use GPL code. It can even handle resource fork stuff if you end up needing it.

See ResourceFile's code for saving as Format::diskimage for an example.

Exactly what I was going to suggest. retro68 even has example code in the Rez implementation that takes a configurable set of files and vacuums them up into a disk image, iirc.

EDIT: Oh hah that's exactly what you linked to.

 

[Phipli]

Not exactly what you're talking about, but I made this a while ago using existing command line tools. The web interface lets you upload a .sit for example, and it hands back a HFS image.

The command line utilities used should do what you need, and... Should be possible to run cross platform... That might be the difficult bit.

GitHub - Phipli/HFSDiskImageWrapper: A little php webpage that allows you to upload a file and recieve it back wrapped in a Classic Mac OS compatible disk image. These images should work straight with the BMOW Floppy EMU or Mini vmac.

A little php webpage that allows you to upload a file and recieve it back wrapped in a Classic Mac OS compatible disk image. These images should work straight with the BMOW Floppy EMU or Mini vmac....

github.com

 

[Phipli]

Also, "genisoimage" is open source, so that actually contains the code you asked for.

 

[LaPorta]

What is the desired outcome for this? Or, what will it make easier?

 

[bigmessowires]

The goal is to make it easier to use StuffIt files with Floppy Emu. Many people are confused about the difference between a disk image and a StuffIt file, because when you grab something off Macintosh Garden you're about equally likely to get one or the other. A disk image can be copied to an SD card and used directly with the Floppy Emu, but a StuffIt file can't. First you need to use some other tool to encapsulate the StuffIt file inside a disk image, like HFVExplorer or Mini vMac. This is too confusing for some people, so they send me grumpy emails or just give up. My hope is to make it so StuffIt files can actually be used like a disk image, if they're small enough to fit on a floppy disk, and the Floppy Emu would automatically convert it to a disk image when you select the StuffIt file from the Emu's disk selection menu. The same trick might be useful for BinHex files too, and maybe some other types.

I'd looked at a bit at libhfs previously, but I was hoping for something simpler than a full-on HFS implementation. After all I'm running this on an 8-bit MCU with a few KB of RAM and tens of KB of free space for new code. I think what I may do is craft my own solution based on Inside Macintosh, looking at the libhfs and genisoimage code to help clarify places where Inside Macintosh isn't very obvious. I'm hoping I can boil it down to something simple that doesn't really understand HFS - it just blindly copies a boilerplate chunk of bytes that represents a pre-made set of HFS boot blocks, master directory block, and catalog with one file entry. Then copies the StuffIt file to a fixed location in the disk image, and patches up a few entries for the volume bitmap and Finder info (name, date, creator and file type).

I'm not sure yet if there are other files and directories that must exist on the disk in order for it to work correctly in the Finder. As I vaguely recall, there are hidden directories for the Desktop and Trash, and some hidden files like DesktopDB, which aren't technically required by HFS but are required in order to keep the Finder happy.

 

[Melkhior]

I've looked for a similar solution to initialize the RAM disk in the NuBusFPGA, and then gave up and just fell back on the 'store an empty FS in the DeclRom and dump it onto the RAM disk'. It's just RLE-compressed and most of the data in the original HFS are just init values that are skipped to save on time.
Unfortunately it doesn't adapt to a different size, but so far it has done the job...

For your use case, an empty floppy image and the hfstools might be all you need? "hmount <img>", "hcopy <file> :", "humount" is probably all that is needed under Linux, I haven't checked if the hfstools are usable in MacOSX and Windows though.

EDIT: following dougg3's post, I also missed you wanted this on the MCU, where hfstools are unlikely to work, sorry.

 

[dougg3]

Ah, I missed the point of what you were trying to do. Didn't realize you were trying to make it work directly on an MCU.

Yeah, I wonder if you could use genisoimage/hfstools/whatever to make an image from a few different file sizes and see if you can notice a simple pattern in the final output...

 

[Snial]

The goal is to make it easier to use StuffIt files with Floppy Emu. Many people are confused about the difference between a disk image and a StuffIt file, because when you grab something off Macintosh Garden you're about equally likely to get one or the other. A disk image can be copied to an SD card and used directly with the Floppy Emu, but a StuffIt file can't. First you need to use some other tool to encapsulate the StuffIt file inside a disk image, like HFVExplorer or Mini vMac. This is too confusing for some people, so they send me grumpy emails or just give up. My hope is to make it so StuffIt files can actually be used like a disk image, if they're small enough to fit on a floppy disk, and the Floppy Emu would automatically convert it to a disk image when you select the StuffIt file from the Emu's disk selection menu. The same trick might be useful for BinHex files too, and maybe some other types.

I'd looked at a bit at libhfs previously, but I was hoping for something simpler than a full-on HFS implementation. After all I'm running this on an 8-bit MCU with a few KB of RAM and tens of KB of free space for new code. I think what I may do is craft my own solution based on Inside Macintosh, looking at the libhfs and genisoimage code to help clarify places where Inside Macintosh isn't very obvious. I'm hoping I can boil it down to something simple that doesn't really understand HFS - it just blindly copies a boilerplate chunk of bytes that represents a pre-made set of HFS boot blocks, master directory block, and catalog with one file entry. Then copies the StuffIt file to a fixed location in the disk image, and patches up a few entries for the volume bitmap and Finder info (name, date, creator and file type).

I'm not sure yet if there are other files and directories that must exist on the disk in order for it to work correctly in the Finder. As I vaguely recall, there are hidden directories for the Desktop and Trash, and some hidden files like DesktopDB, which aren't technically required by HFS but are required in order to keep the Finder happy.

Wouldn't it be easier just to experiment with reverse engineering Floppy Disk images containing a single stuffit file of various sizes? HFS has an underlying B*-Tree and tries to create files with contiguous blocks doesn't it? So, it should be fairly easy to see how a single file is encoded. There's probably then a sequential relationship between the stuffit file's disk blocks and the disk image?

Also, what if Floppy Emu is being used with a pre-Mac Plus? Wouldn't you need to encapsulate a stuffit file as an MFS disk?

 

[Phipli]

The goal is to make it easier to use StuffIt files with Floppy Emu. Many people are confused about the difference between a disk image and a StuffIt file, because when you grab something off Macintosh Garden you're about equally likely to get one or the other. A disk image can be copied to an SD card and used directly with the Floppy Emu, but a StuffIt file can't. First you need to use some other tool to encapsulate the StuffIt file inside a disk image, like HFVExplorer or Mini vMac.

Well... That's exactly what my tool I shared the GitHub link for does. I got sick of people complaining that they could use .sit files on their floppy emus back when I used Facebook Mac groups, I made that and hosted it on my NAS so that I could just take their files and encapsulate them in an image in seconds.

Feel free to improve it and host it somewhere if you like. Limitations are mainly that the server upload size needs setting to something sensible.

I'd be hosting it myself except I wanted someone to cast their eye over the security and my web host doesn't give me access to the CLI.

 

[Phipli]

As I vaguely recall, there are hidden directories for the Desktop and Trash, and some hidden files like DesktopDB, which aren't technically required by HFS but are required in order to keep the Finder happy.

If they're not there it just creates them I think.

 

[eharmon]

I wonder if you could pre-generate a fixed filesystem with a giant empty contiguous file the size of the filesystem, then at runtime patch a few hardcoded offsets in the HFS metadata which define the file extent's size to match the stuffit and redirect reads to the hardcoded block range where the file is to the stuffit on disk. I think you'd have to patch a few more things to avoid "corruption" as the truncation would create lost blocks that were allocated in the file table but not used by the file extent anymore. But realistically that probably wouldn't be a big deal since it's a read-only filesystem. Not like those lost blocks cause any problems.

Then you'd just need to store the 1k or whatever of filesystem metadata.

 

[LaPorta]

Now I get exactly what you are doing, and that is an excellent idea. Something similar is done for arcade machine ROM files: they are stored as .zips and the program seamlessly reads the ROM files inside the archive.

How would you account for the different, incompatible StuffIt files?

 

[cheesestraws]

I wonder if you could pre-generate a fixed filesystem with a giant empty contiguous file the size of the filesystem, then at runtime patch a few hardcoded offsets in the HFS metadata which define the file extent's size to match the stuffit and redirect reads to the hardcoded block range where the file is to the stuffit on disk. I think you'd have to patch a few more things to avoid "corruption" as the truncation would create lost blocks that were allocated in the file table but not used by the file extent anymore. But realistically that probably wouldn't be a big deal since it's a read-only filesystem. Not like those lost blocks cause any problems.

Yeah, this feels like a very sensible approach.

 

[Snial]

Now I get exactly what you are doing, and that is an excellent idea. Something similar is done for arcade machine ROM files: they are stored as .zips and the program seamlessly reads the ROM files inside the archive.

How would you account for the different, incompatible StuffIt files?

The files don't get unstuffed, it's just the literal file embedded in a disk image.

 

[LaPorta]

Ok I thought this was going to give you direct file access, skipping not only the disk image, but that “what to do if I’m new and don’t have StuffIt yet” issue.

 

[bigmessowires]

Well... That's exactly what my tool I shared the GitHub link for does. I got sick of people complaining that they could use .sit files on their floppy emus back when I used Facebook Mac groups, I made that and hosted it on my NAS so that I could just take their files and encapsulate them in an image in seconds.

That's a great tool. I'm trying to do essentially the same thing, except as a built-in behavior of the device.

Wouldn't it be easier just to experiment with reverse engineering Floppy Disk images containing a single stuffit file of various sizes?

I wonder if you could pre-generate a fixed filesystem with a giant empty contiguous file the size of the filesystem, then at runtime patch a few hardcoded offsets in the HFS metadata which define the file extent's size to match the stuffit and redirect reads to the hardcoded block range where the file is to the stuffit on disk.

I think you're both suggesting something similar, and yes I think that's a good approach. I'll look into it more.

Also, what if Floppy Emu is being used with a pre-Mac Plus? Wouldn't you need to encapsulate a stuffit file as an MFS disk?

I'd thought about this, and yes it could be done, but I think it'd only be useful in rare cases. HFS was introduced in System 2.1, so every Macintosh model is HFS-capable. And in my experience the very oldest Mac software is usually archived as disk images, not as StuffIt files. So it would only be an issue if you're trying to transfer a Stuffed file to your System 1 startup floppy. A more practical problem for those oldest Macs is that if your computer doesn't have a hard drive, then you'd need one floppy with a bootable System file, a second floppy with the StuffIt application, and a third floppy (from Floppy Emu) with the StuffIt file.

Ok I thought this was going to give you direct file access, skipping not only the disk image, but that “what to do if I’m new and don’t have StuffIt yet” issue.

That would be very slick! But no, that's probably outside the bounds of what I can hope to do on the device, and it would also result in larger disk images that would be more likely to exceed 800K or 1.4MB. I really just want to provide a way for people to easily get a .sit file onto their Mac's hard drive, so they can un-Stuff the file there, since it's a common use case that's not directly supported.

 

[NJRoadfan]

You may want to look at Andy McFadden's CiderPress2: https://github.com/fadden/CiderPress2

He has notes on HFS here: https://github.com/fadden/CiderPress2/blob/main/DiskArc/FS/HFS-notes.md

Most of the focus is on Apple II related formats, which HFS falls under since GS/OS can use those volumes.

 

[bigmessowires]

Sort-of related: I'm also working on adding Floppy Emu support for ISO and Toast CD images, and also for partitioned SCSI device images like the ones used with Zulu SCSI and Blue SCSI. This is mostly working now, and just needs a little bit of clean-up. For images that contain a partition map, Floppy Emu will parse the partition map and mount the first Apple_HFS partition that it finds as an HD20-type hard drive. ISO, CDR, and TOAST images are mounted as read-only, while partitions from SCSI emulator HDA and IMG images are mounted as read-write. It's not going to replace a SCSI emulator, since the I/O speed is slow and it's only available for Mac models with HD20-type disk support, but it works and is pretty handy! It's fun to see a Myst CD or CodeWarrior CD mounted on my Mac IIci desktop via Floppy Emu.

I did run into one issue where I could use a sanity check: do CDs essentially not work under System 6? If I mount the HFS filesystem from an ISO as a writable volume, System 6 complains the disk is "so full that the desktop file can't be created", then prompts me to make minor repairs, then complains about the desktop file again in an endless loop. Or if I mount the filesystem as a read-only volume, System 6 complains that I need to unlock the disk so that it can create the desktop file, and when I dismiss the dialog it immediately reappears. So either way, it can't be used. The only exception I've found is Myst, which doesn't trigger this error, but every other ISO and TOAST file seems to have this problem. Under System 7 it works fine.