ISO sample code to create an HFS filesystem or disk image

[LaPorta]

So, the latter StuffIts (I can’t say exactly when), and Stuffit Expander started coming with the Stuffit Engine Extension and such, it started becoming a Swiss Army knife and could open mostly anything. It won’t be able to open the sitx variety, though I think the early Stuffit 6/7 that ran on both OS X and 9 via Carbon can.

Long story short, the StuffIt Expander 5 that is probably the most common that works on System 7/8/9 will open 1.5.1 and latter ones that would be seen in a normal OS environment, excluding the stuff made on OS X. That includes ones without type/creator codes or file extensions. It just somehow knows how to open them.

 

[LaPorta]

I also agree with above: we could
Probably make a few archives and then just get the type/creator codes via ResEdit.

 

[bigmessowires]

Probably make a few archives and then just get the type/creator codes via ResEdit.

Yes, but using which version of StuffIt to make the archives? That's the tricky bit. I'll do some experiments as you suggest.

I had a flash of inspiration for another related Floppy Emu feature that I think would be great. In short, it's the opposite of the feature I've described thus far. Instead of taking a file on the FAT32 SD card and automatically wrapping it in an HFS disk image, it would take an HFS disk image on the SD card and "explode" it into a directory of individual files on the SD card. Armed with all the HFS knowledge that I've gained in the past couple of days, I'm pretty sure I could do this.

This would be super useful when you have an important file on your vintage Mac, like your 1990 Master's thesis document in Word 6 format, and you want to use Floppy Emu to transfer it to your Windows 11 PC. That's possible now, but it's cumbersome: You need to save the document to a disk image on the Floppy Emu, then use another PC-hosted tool like Mini vMac or HFVExplorer to mount the disk image and extract the file. From talking with customers over the years, this is too challenging and confusing for some people to manage. But what if you could simply choose an 'extract files' option directly on the Floppy Emu and have it dump all the files from a disk image straight to the SD card? Then there wouldn't be a need for Mini vMac or HFV Explorer - you'd just have the file you wanted right there.

This would need to handle resource forks in an intelligent way. For a file "Foo" that has both data and resource forks, I'm thinking it would write Foo.data and Foo.rsrc to the SD card. If there were no resource fork, then it would simply write out Foo. A reverse mapping table of creator/type IDs to filename extensions could also be used, so "Thesis" with MSWD/W6BN creator/type would be written out as Thesis.doc.

 

[LaPorta]

My suggestion was using different versions to make various archives…that way you can compare/contrast. Sorry I didn’t spell that out.

 

[Phipli]

StuffIt I think there are several different ones depending on the StuffIt versions, but I'm not sure whether it matters.

You can use ! As a version number wildcard with... Some / all stuffit files. I forget the codes / capitalisation without booting a machine to check, but effectively "SIT!" Instead of "SIT5" or whatever.

 

[NJRoadfan]

Yes, but using which version of StuffIt to make the archives? That's the tricky bit. I'll do some experiments as you suggest.

I had a flash of inspiration for another related Floppy Emu feature that I think would be great. In short, it's the opposite of the feature I've described thus far. Instead of taking a file on the FAT32 SD card and automatically wrapping it in an HFS disk image, it would take an HFS disk image on the SD card and "explode" it into a directory of individual files on the SD card. Armed with all the HFS knowledge that I've gained in the past couple of days, I'm pretty sure I could do this.

For Apple II images I have been using CiderPress to read/write to images. Andy has a system that appends an extension to extracted files with the Filetype/auxtype (The Apple II equivalent of Filetype/Creator Type metadata) See: https://nulib.com/library/nulib2-preserve.htm

For resource fork data, is a dot file (OS X style) with an AppleDouble header possible?

 

[bigmessowires]

Yes, a tool like CiderPress or HFVExplorer will be the best solution for extracting files, and I wouldn't try to replace those tools. This would be more like an in-emulator version of the simplest use case of those tools, to save time and hassle. There are many options for how the extracted HFS files might be stored on the SD card: .data and .rsrc files, or as a single MacBinary .bin, or maybe something else similar to how CiderPress does it. I'm not familiar with AppleDouble, but I'll read about it. It sounds similar to what I was proposing, except instead of Foo.data and Foo.rsrc, you would have Foo containing the data and ._Foo containing the resources and Finder metadata. Maybe this is a bigger feature than I initially imagined, and best saved for another time.

 

[cheesestraws]

Can I suggest starting with just supporting data fork only files? As soon as you also get a resource fork, wrangling the fs will become harder, and even a data-fork-only option will be useful (let's face it, people use FTP all the time and that's data fork only).

In the interests of sanity, personally I'd file AppleDouble and resource forks as a "later" problem, once the simpler data-fork-only case has been done and shown to be reliable/useful.

 

[LaPorta]

So in my tests so far, Stuffit 1.3.1 and 1.5.1 both Produce file Type SIT!. Stuffit 5 makes SIT5 file types. Stuffit 8 makes SITX file types (no surprise there), along with the .sitx extension. Stuffit Expander 5.5 can handle any of the original types.

Each and every one has a creator code of SIT!

 

[bigmessowires]

I'm having some trouble understanding how the current versions of MacOS store resources. AppleDouble stores Filename's resources in a separate file named ._Filename, which seems simple enough.

Confusingly, it appears that OSX only uses AppleDouble when storing files on a non-Mac filesystem like FAT32 or EXFAT. Otherwise it does something completely different and stores resources in a subdirectory? Wikipedia's info says forks are named file/..namedfork/forkname, e.g., the resource fork of the file IMG_0593.jpg is IMG_0593.jpg/..namedfork/rsrc.

If I'm generating my own AppleDouble files, are there any OSX tools I could use to view/edit them, to confirm that my files are constructed correctly? Basically what's the OSX equivalent of Res Edit, or something simpler that would just parse and list the resources? There's a tool called ResForge that looks like it might be what I'd want?

Another thought: if I'm going to support wrapping individual files into an HFS disk image, and I'm also going to support exploding a disk image into individual files, then those two features need to interoperate. If I explode a disk image and create a file with separate data and resource parts, then wrapping that same file needs to identify and include the resource parts in the HFS wrapped disk image.

@cheesestraws may be right that resources are best left for later, but I'd like to do something with the resources beyond just ignoring them, even if it's not perfect. But I also need to remember that the main use case for wrapping files is compressed archives, which don't have a resource fork, and the main use case for exploding disks is probably to extract word processing or text files or images, which normally don't have a resource fork either. And for more complex needs there are other stand-alone tools, so I don't need to do everything here. I only need to cover the basics.

A last point - Although OSX lost the ability to read/write HFS disks some time ago, it's still possible (or was still possible) to mount HFS disk images as read-only. If you have a modern Mac, that's a quick and easy way to extract files, and a better alternative to the proposed "explode disk" feature for the Floppy Emu. The Floppy Emu instruction manual section 8.2 describes how to mount vintage Macintosh HFS disk images as read-only on OSX. But when I tried it just now (running MacOS Ventura 13.4), it only complains "no mountable filesystems" or "the disk you attached was not readable by this computer". Can I assume this capability was removed? When did that happen?

 

[LaPorta]

Yes, just a few system revisions ago, the ability to even read HFS was stripped out. FuseHFS is the only way I know of to mount such images now.

 

[cheesestraws]

Wikipedia's info says forks are named file/..namedfork/forkname, e.g., the resource fork of the file IMG_0593.jpg is IMG_0593.jpg/..namedfork/rsrc.

This isn't a real directory, this is an extension to the UNIX path syntax to be able to address arbitrary forks of a file by making a file with multiple forks look kind of like a directory. It's a bit confusing but I have to admit I can't think of anything better myself.

But I also need to remember that the main use case for wrapping files is compressed archives, which don't have a resource fork, and the main use case for exploding disks is probably to extract word processing or text files or images, which normally don't have a resource fork either. And for more complex needs there are other stand-alone tools, so I don't need to do everything here. I only need to cover the basics.

Yeah, this is part of why I suggested that. Also - adding a resource fork is a lot like adding a second file, so so long as you don't make any hard assumptions that the disc images you're generating only contain a single file, the resource fork-less version is a useful stepping stone towards a resource-fork-handling version - it's not like it's throwaway work, it's more like an intermediate stage that is itself useful.

When did that happen?

Went in 10.15.

 

[bigmessowires]

The new Floppy Emu feature is finished and working well to wrap StuffIt, HQX, and similar files inside a temporary floppy disk image. From the user's point of view, files like .sit and .hqx can now be used exactly like .dsk and .img disk images. I wish I'd thought of this a long time ago! For the way that I normally use my Floppy Emus, this is a major time-saver and quality of life improvement. I'm calling this process "boxing", as in placing the archive file inside a box.

I've also made some progress on the reverse feature which "explodes" a disk image into individual files, which can then be retrieved directly from the SD card. My preliminary code is working, but with several limitations, and I'm gradually realizing that exploding an arbitrary disk image is much harder than synthesizing a disk image to box up one file. I've decided to set aside exploding for the moment, and concentrate on finishing off and releasing the other new features. Hopefully I'll have a new firmware release in the next day or two.

Some notes on exploding disk images: It requires understanding and traversing the B-tree nodes in the catalog file. The catalog file might be fragmented, or the data files themselves might be fragmented. For the moment I'm only supporting a maximum of three fragments for a file, which is the limit before the "extents overflow" file needs to get involved - yet another B-tree. In some testing of real disks, I only found two files that had more than three fragments, so I don't think this is a major limitation.

There's work to do on filename conversion. While I've largely ignored the differences between Mac and FAT32 legal filename characters when boxing a StuffIt archive into a disk image, I can't really ignore it in the other direction, because non-ASCII characters are so common in Mac filenames. I've encountered lots of folder names with the script-f character, files with the (tm) or (R) symbol in their names, etc.

The bigger challenge is replicating the directory structure. While traversing the catalog file, I haven't found a simple way to determine the full path to a file. The only info available is that you have a file named "Chicago" in directory 34. Then you need to scan the catalog file again to determine that directory 34 is "Fonts" and is in directory 28. Then scan again to determine that directory 28 is "System Folder" and is in directory 2. Then scan again to determine that directory 2 is "Disk Tools" and is the root directory. This isn't so bad on a modern computer, but it's difficult on an 8-bit microcontroller with limited RAM. It's slow and it requires additional RAM for every level of directory depth, which is potentially unbounded. For the time being I've implemented a simple directory ID:name cache with 64 entries, which is enough for exploding a floppy disk but probably insufficient for larger disk images. Maybe there's a smarter algorithmic approach; I'll give it some thought. I need to avoid recursive algorithms because there just isn't enough RAM for a bunch of stack frames.

One potential work-around might be to limit the exploding to files in the root directory only. So if you wanted to export your Master's thesis Word document, you'd need to copy it to the root directory of a floppy disk image first. That would work, but wouldn't be very intuitive or user-friendly.

Another work-around might be to flatten the directory structure when exploding a disk. All of the files from all of the disk's subdirectories would get exported, but they'd all be stored in a single directory on the SD card. This would create the possibility of having multiple files with the same name in the same directory, which could maybe be addressed by adding automatic filename suffixes like (1) and (2). For a floppy disk this approach might be acceptable, but for a larger disk it would be a mess to see every single file dumped into a single shared directory.

A third possibility might be to implement a GUI browser where you could directly explore inside the disk image via the Floppy Emu display and buttons, and select one specific directory to export. But... that seems like a lot of work.

Other suggestions are welcome! I think the main use case for this feature would be exporting a single word processing or text document of personal historical interest. A second likely use case might be exporting all the source code files from an old programming project. I can't immediately think of any other situations where you'd urgently want to get some files off of your 25-year-old disk. So solutions that are tailored towards these particular use cases are probably best. For more complex needs, there's always Fuse HFS, HFVExplorer, Mini vMac, etc.

 

[Phipli]

StuffIt, HQX, and similar files inside a temporary floppy disk image

Does that include macbinary? Usually a .bin.

 

[Phipli]

One potential work-around might be to limit the exploding to files in the root directory only. So if you wanted to export your Master's thesis Word document, you'd need to copy it to the root directory of a floppy disk image first. That would work, but wouldn't be very intuitive or user-friendly.

Could you add a special "virtual" image for exporting files to the raw SD card? Mount it, then copy a file to it and it vanishes into the ether... written to a folder on the actual SD card as a MacBinary or similar.

The virtual disk could be called the black hole or SDrd Out or something.

 

[bigmessowires]

Does that include macbinary? Usually a .bin.

Yes, it currently supports sit, hqx, bin, cpt, and txt files. Other types could be added fairly easily.

 

[LaPorta]

As with @Phipli ’s suggestion, perhaps you could call it the “Alternate Disk” after a feature in the early 128k manuals that never made it to production.

 

[cheesestraws]

The new Floppy Emu feature is finished and working well to wrap StuffIt, HQX, and similar files inside a temporary floppy disk image. From the user's point of view, files like .sit and .hqx can now be used exactly like .dsk and .img disk images. I wish I'd thought of this a long time ago! For the way that I normally use my Floppy Emus, this is a major time-saver and quality of life improvement. I'm calling this process "boxing", as in placing the archive file inside a box.

This is a really major useful feature. Thankyou!

 

[bigmessowires]

This is a really major useful feature. Thankyou!

Could you add a special "virtual" image for exporting files to the raw SD card? Mount it, then copy a file to it and it vanishes into the ether... written to a folder on the actual SD card as a MacBinary or similar.

I like the idea of using a special floppy disk image for exporting files, instead of an "explode" feature that can operate on any arbitrary disk image. I think that'll be clearer and easier for the user, and will simplify some things for me too since I can rely on the fact that it's a floppy disk and that it has a specific layout that I control. But I don't think I can export files on-the-fly as they're written to the disk image. It needs to be a post-process that operates on the whole image, once you're done copying files to it.

Something like this might work:

1. The user selects a new item from the Floppy Emu menu: "Insert Exporter Disk..."
2. A sub-menu appears asking if the disk should be 400K, 800K, or 1440K
3. A new temporary disk image temp.dsk is created and saved to the SD card.
4. Temp.dsk gets mounted. The name of the disk is "Copy Files Here". Disk emulation proceeds the same way as for any other disk image.
5. The user copies files from their hard drive to the Copy Files Here disk. Files need to be small enough to fit on a floppy disk.
6. When the Mac ejects the disk, the export process will automatically begin. It'll use something like my work-in-progress code, with limitations on file fragmentation, total number of directories, and max directory depth, but for a freshly-made floppy it should work 99+ percent of the time. If it fails, it'll show a graceful error message and suggest using a different tool. Otherwise it'll write all the files to a set of directories under the "export" top-level directory on the SD card. Resource forks will be written as AppleDouble or MacBinary or just a chunk of bytes with a .rsrc filename extension, and data forks will be written to regular files.
7. temp.dsk gets deleted
8. The Floppy Emu will display an export report, something like "Successfully exported 8 directories, 26 files".
9. Floppy Emu then returns to the normal disk selection menu

As inspiration for this feature, I'll tell you story. A few years ago I was contacted by a retired scientist from India. Years earlier, he'd written software on a Mac SE that did some very specific complex calculations related to something like light scattering in different materials. The program was written in a text-based scripting language. He still had the Mac SE, but he no longer had a copy of the software and he really wanted to get that program script off his SE. But for a scientist he was not technically savvy, and I think he was getting on in years. I needed to hold his hand through the process of selecting and mounting a blank disk image, copying his program to it, then mounting the SD card on his Windows PC, downloading and running HFV Explorer which isn't the most intuitive software, and using HFVExplorer to extract the script from the disk image. It was painful. A feature like this would make that process much easier for the uninitiated, even if it's not really doing anything you couldn't do already with other tools.

 

[bigmessowires]

Well, I've decided not to pursue the explode disk / export files feature. The straw that broke the camel's back was the discovery that my FAT code only supports creation of files with 8.3 filenames, and the code can't easily be replaced wholesale, so I spent a couple of hours reading about FAT internals and how long filenames are implemented before deciding to call it quits. Even though a feature to export files would be very useful to some people, it's not a common enough need to justify all the work I can see it would require and all the limitations that would come with doing it directly on the Floppy Emu.

Instead, I think what the vintage Mac community really needs is a stand-alone cross-platform application like a simplified version of HFVExplorer. A simple GUI that lists all the files in an HFS disk image and lets you drag files in and out of its window. The GUI might look something like 7Zip, although it would not do any compression, just HFS translation.

It's major overkill to run a full-on Mac emulator like Mini vMac just for the purpose of transferring a few files on or off an HFS disk.

HFVExplorer is OK, but it only runs on Windows, and its interface is somewhat confusing and antiquated.

FuseHFS/macFUSE only runs on a Mac, and it's a filesystem plugin framework rather than a stand-alone application. Just installing it is beyond the capabilities of some people.

HFSExplorer might be a candidate... I haven't tried it. It's written in Java so it's theoretically cross-platform, but the documentation mentions "some Windows-specific parts written in C". It requires a separate installation of JRE 5.0+ which is another stumbling block, and simply launching the program may be a challenge for some people since there's no friendlly application icon to double-click; you need to open a terminal window and run a shell script from the bin directory. I looked at some screen shots of the interface and it seems very similar to HFVExplorer with all its confusing and legacy-styled UI. But despite its warts, this still might be the best option.

Maybe better options already exist and I don't know about them?