Netatalk 4.0 - Future-proofing Apple File Sharing

[NJRoadfan]

You don't run ad cp at all. Copy/extract the files into the share folder like they are standard UNIX files, then run ad set to set the filetype/creator, which forces creation of the Netatalk metadata EA. After that, Netatalk will update the CNID database on access, running dbd is optional.

The problem I had with pax is that it would extract NetBoot.pax to a folder that resulted in "permission denied" when trying to switch to it despite being owned by the user who extracted them!

 

[saybur]

Copy/extract the files into the share folder like they are standard UNIX files, then run ad set to set the filetype/creator, which forces creation of the Netatalk metadata EA. After that, Netatalk will update the CNID database on access, running dbd is optional.

I tried this again and unfortunately ran into the same issue. Commands ran on an empty share with dbd -f /srv/netboot ran on it immediately beforehand (lightly edited for brevity):

$:/srv/netboot$ 7z x ~/NetBoot.pax
$:/srv/netboot$ chmod +x NetBootInstallation
$:/srv/netboot$ cd NetBootInstallation
$:/srv/netboot/NetBootInstallation$ ad set -t dimg -c ddsk "Applications HD.img"
$:/srv/netboo/NetBootInstallationt$ ad set -t dimg -c ddsk "NetBoot HD.img"
$:/srv/netboot/NetBootInstallation$ ad set -t chrp -c tbxi "Mac OS ROM"
$:/srv/netboot/NetBootInstallation$ sudo dbd -f /srv/netboot

Each ad returned something like {netatalk_conf.c:2001} (error:Default): getvolbypath("/srv/netboot/NetBootInstallation/Applications HD.img"): path does not match basedir regex "/home". I saw no activity in Netatalk's debug log for these commands. The final dbd produced:

Bad AppleDouble "/srv/netboot/NetBootInstallation/._NetBoot HD.img"
Bad AppleDouble "/srv/netboot/NetBootInstallation/._Mac OS ROM"
Bad AppleDouble "/srv/netboot/NetBootInstallation/._Applications HD.img"

Share definition:

[NetBootVol]
path = /srv/netboot
valid users = netboot

(Note user here is not netboot, in case that is a potential source of problems)

 

[saybur]

Made another attempt, this time within the ~/afp-data directory. This time I got "ad[58969] {unix.c:619} (error:AFPDaemon): initgroups(saybur, 1000): Operation not permitted". Adding sudo to each ad command made them complete without error and I did see a burst of activity in the Netatalk log for each invocation. At the end of the process I ran sudo dbd -f ~/afp-data/ and got the following:

Bad CNID in adouble file of '/home/saybur/afp-data/NetBootInstallation/Mac OS ROM'
CNID mismatch for '/home/saybur/afp-data/NetBootInstallation/Mac OS ROM', CNID db: 18, ad-file: 0
Bad AppleDouble "/home/saybur/afp-data/NetBootInstallation/._NetBoot HD.img"
Bad AppleDouble "/home/saybur/afp-data/NetBootInstallation/._Mac OS ROM"
Bad CNID in adouble file of '/home/saybur/afp-data/NetBootInstallation/Applications HD.img'
CNID mismatch for '/home/saybur/afp-data/NetBootInstallation/Applications HD.img', CNID db: 19, ad-file: 0
Bad AppleDouble "/home/saybur/afp-data/NetBootInstallation/._Applications HD.img"
Bad CNID in adouble file of '/home/saybur/afp-data/NetBootInstallation/NetBoot HD.img'
CNID mismatch for '/home/saybur/afp-data/NetBootInstallation/NetBoot HD.img', CNID db: 20, ad-file: 0

Subsequent runs just had the "Bad AppleDouble" entries.

I'm out of time tonight but can give it another go tomorrow if I'm doing something obviously wrong here.

 

[NJRoadfan]

Don't worry about the "bad CNID" error messages, those are normal for files added from outside the share and the dbd tool is fixing them.

The "Bad AppleDouble" message is also from dbd. Looks like it only throws that with the ._ files. They likely aren't 100% like what Netatalk expects since it didn't generate them (The check in ad_valid_header_osx() fails). There is extra filler text in Netatalk ._ files starting at byte 08 after the AppleDouble header. This string should be 16 bytes starting with the string "Netatalk" and padded with spaces (0x20). The ._ files from OS X lack this but this can be "fixed up" with a script or some Python code. Accessing the files via AFP should product the desired results regardless, resource forks should be intact.

The initgroups error message when running the ad tools in a shared home directory is a long standing bug.

 

[saybur]

Appreciate the help, I will try these out via a client later this evening.

 

[NJRoadfan]

Reading thru the source, it appears that if a ._ file has extended attributes embedded in it, that Netatalk will indeed copy the FinderInfo from that file and into a metadata EA if one isn't found. It will also re-write the ._ file minus the embedded extended attributes (discards them) with the proper header filler text. In other words, it properly converts the data automatically without the need for "ad set".

It won't do this for otherwise ordinary ._ files without extended attributes like the NetBoot images have. This is likely due to performance and security concerns.

As for the embedded extended attributes getting tossed, its likely because nobody felt to need to write a parser to restore them to the parent file.

 

[NJRoadfan]

In addition to the ad set commands, add the following to your script to "fix up" the header of your ._ files so Netatalk stops complaining about bad AppleDouble data:

printf '\x4E\x65\x74\x61\x74\x61\x6C\x6B\x20\x20\x20\x20\x20\x20\x20\x20' | dd of="._Applications HD.img" bs=1 seek=8 count=16 conv=notrunc
printf '\x4E\x65\x74\x61\x74\x61\x6C\x6B\x20\x20\x20\x20\x20\x20\x20\x20' | dd of="._NetBoot HD.img" bs=1 seek=8 count=16 conv=notrunc
printf '\x4E\x65\x74\x61\x74\x61\x6C\x6B\x20\x20\x20\x20\x20\x20\x20\x20' | dd of="._Mac OS ROM" bs=1 seek=8 count=16 conv=notrunc

This is quick and dirty, but can easily be converted into a loop or subroutine. While Netatalk works with these files without the above edits, it will not hide the ._ files from the client unless the "filler" has the string "Netatalk " (8 spaces) in it. The files straight from the archive has all null bytes here. You could go further and use other tools to programmatically extract the filetype/creator from the ._ file and apply them to the Netatalk metadata EA.

 

[saybur]

In addition to the ad set commands, add the following to your script to "fix up" the header of your ._ files so Netatalk stops complaining about bad AppleDouble data:

Very nice, that's cleaner than the manual bless editing I was doing. This and the set commands gets EAs correctly populated with FinderData and I'm no longer getting AppleDouble errors on dbd scans.

Unfortunately this is not bringing over the resource fork from the dot files into the EAs, and watching the debug log during fork requests, it looks like Netatalk isn't pulling it from the dot files directly either, if that's the intended behavior.

I did find a workaround but it's a kludge:

1. Starting with a new folder, copy only the main files (no dot files) and set ea = ad on the share.
2. Run dbd -f to generate an .AppleDouble folder with "blank" AD files in the format Netatalk expects.
3. Manually insert FinderData and the resource fork into those files.
4. Re-run the database command (not sure if this step is strictly required).

This gets the resource fork seen and the client is able to netboot from the share. The code I wrote for this isn't elegant but it appears to work OK.

For my part I'm good with this as a solution. While something more "out of the box" would be awesome I'd rather not have you and @slipperygrey spend time on what seems like a pretty niche use case (but I do greatly appreciate the assistance from both of you!)

 

[NJRoadfan]

OK, the above will only work if you are using ea = sys and NOT the old AppleDouble v2 system with ea = ad. IT should be pulling the resource fork from the ._ files as I have it working here. You need both files in the directory together, Netatalk always stores the resource fork in the ._ file.

 

[saybur]

OK, the above will only work if you are using ea = sys and NOT the old AppleDouble v2 system with ea = ad.

Sorry, I should have clarified that all original testing was with ea unset; I was checking extended attributes directly with addump -e (and to a lesser extent getfattr --dump, just to make sure something was present). I went back to ea = ad as a fallback since it was somewhat easier to directly manipulate the files (and that was only once I started down the "do it manually" route).

 

[NJRoadfan]

Since this is on-topic for a Netatalk thread, here are the nitty gritty details on what is stored with each AppleDouble backend:

The "old" AppleDouble v2 system:
All metadata for files is stored in a separate hidden directory called .AppleDouble with equivalently named files. These files follow the AppleDouble format as outlined by Apple and store the following:

• File Name ("Long" name)
• Finder Comments
• File Date Information
• FinderInfo (filetype/creator, label, etc.)
• DID (unused by Netatalk)
• AFP File Info (additional AFP specific file attributes)
• "Short" File Name (unused by Netatalk, for MS-DOS 8.3 files)
• ProDOS Info (unused by Netatalk, generated on the fly by afpd)
• UNIX Dev ID (Netatalk Private Field)
• UNIX Inode ID (Netatalk Private Field)
• UNIX Syn ID (Netatalk Private Field)
• CNID of file (Netatalk Private Field)
• Resource Fork

The new extended attributes system:
With this system, metadata is split into two places, an extended attribute in each file called org.netatalk.Metadata, and if the file has a resource fork, a matching file prefixed with "._". On Solaris systems with zfs, this file won't be present and the resource fork is stored as an extended attribute called org.netatalk.ResourceFork. On Linux, these EAs will be prefixed with "user."

The metadata EA stores the following information in AppleDouble format:

• FinderInfo (filetype/creator, label, etc.)
• Finder Comments
• File Date Information
• AFP File Info (additional AFP specific file attributes)
• UNIX Dev ID (Netatalk Private Field)
• UNIX Inode ID (Netatalk Private Field)
• UNIX Syn ID (Netatalk Private Field)
• CNID of file (Netatalk Private Field)

The ._ file stores the following information in AppleDouble format conforming to Apple's spec for these files:

• FinderInfo (filetype/creator, label, etc.) This data isn't used by Netatalk, it is only pulled from the metadata EA.
• Resource fork

If running Netatalk 4.1 or later on macOS, the Resource fork and FinderInfo are natively stored in the file system and used. If setting up a new Netatalk installation, this is the preferred back end since it can be made to interoperate with Samba, and has proper macOS integration.

 

[slipperygrey]

@NJRoadfan what would you say is the expected behavior for resource fork EAs when running a non-Solaris OS with ZFS, say FreeBSD? Is the org.netatalk.ResourceFork EA a Solaris extension?

 

[saybur]

Since this is on-topic for a Netatalk thread, here are the nitty gritty details on what is stored with each AppleDouble backend:

This is great info thank you, I was not understanding this prior to your posts on the subject. If it was in the documentation I definitely missed it, adding that somewhere in the official docs would be useful.

I also find this unintuitive, to be honest. I assume there's a technical reason not to put everything into extended attributes but having components of a file split across 3 different locations feels... icky.

@NJRoadfan what would you say is the expected behavior for resource fork EAs when running a non-Solaris OS with ZFS, say FreeBSD? Is the org.netatalk.ResourceFork EA a Solaris extension?

FWIW I use OpenZFS on Linux so this question might be relevant there was well.

 

[NJRoadfan]

The org.netatalk.ResourceFork EA is only used in Solaris. The extended attribute API that works similar to file descriptors (complete with random access) only exists on that OS. Even Darwin doesn't have this, although with some coding, you can simulate it for the resource fork (some of Apple's tools do this). Its easier to use the POSIX /..namedfork/rsrc abstraction though.

Generally, extended attributes are very small, and have an upper size limit on most file systems (usually 4kb in case of ext4). ZFS treats them as files, so no upper limit there. The APIs to manipulate them (on just about everything except Solaris) assume reading and writing an entire EA in one pass since they are usually only a few bytes big.

The storage of data in two places is technically icky, but needed for compatibility reasons. Netatalk can't add its needed internal fields to the ._ files since Apple only specced them to contain FinderInfo and and resource fork.

 

[saybur]

The storage of data in two places is technically icky, but needed for compatibility reasons. Netatalk can't add its needed internal fields to the ._ files since Apple only specced them to contain FinderInfo and and resource fork.

That makes complete sense, thanks.

Are there any plans to change the old-school ea=ad approach? Or is that expected to remain a viable option for the future?

 

[NJRoadfan]

The old ea=ad system isn't going away anytime soon.

 

[slipperygrey]

As a concrete example, OpenBSD has chosen not to provide EA capabilities in their default file systems (FFS, FFS2) and probably never will. I don't have any intention to remove AppleDouble v2 from netatalk.

 

[pl212]

Apologies that I don't quite know whether to post this in the TashTalk thread, the TashRouter thread, or here.

I have a TashTalk hat running on the Raspberry Pi that runs the latest version of Netatalk, 4.3.2. Would it be best practice to continue to use the tap and python script method detailed here, or is there a way to combine the networks using only Netatalk's own atalkd routing features?

My use case is bridging the physical LocalTalk network of the TashTalk hat with the physical Ethernet network that the Pi is also connected to. It strikes me having an atalkd-only method, if possible, would simplify deployment and debugging...

Thanks for any thoughts!

 

[NJRoadfan]

Since atalkd doesn't support the TashTalk directly, you'll have to run both regardless. In my case, I create a tap interface (called tash0) and "connect" both atalkd and tashrouter to it with a dedicated network number. This avoids using kernel bridging and some other weird side effects, like TashRouter not seeing packets coming from the kernel itself when Netatalk services attempt to communicate. This adds an extra routing hop, but in practice doesn't add much overhead.

 

[shirsch]

Since atalkd doesn't support the TashTalk directly, you'll have to run both regardless. In my case, I create a tap interface (called tash0) and "connect" both atalkd and tashrouter to it with a dedicated network number. This avoids using kernel bridging and some other weird side effects, like TashRouter not seeing packets coming from the kernel itself when Netatalk services attempt to communicate. This adds an extra routing hop, but in practice doesn't add much overhead.

For those of us who are continually confounded by bridges, tunnels, taps, etc. would you consider posting a complete example? I'm planning to stand up a new tashrouter + netatalk server and could use just a wee more concrete guidance.