Netatalk 4.0 - Future-proofing Apple File Sharing

[slipperygrey]

I've been going through the 3.x afpd code base seeing what needs to be restored. Eventually I'll get my notes up, but there are two parts of this so far:

1. Adding back pre-AFP 2.2 compatibility: Thankfully most of this is still in place in the code base. The biggest change is restoring the ProDOS and free space size limiting volume flags from 2.x.

2. Adding back DDP/ATP transport support: This is going to be where most of the work needs to be done. The 2.x code base spawns separate child process types for DDP/ATP or TCP/DSI depending on the connection. In 3.0, most of this was simplified and hard coded for TCP/DSI.

The PR that brings back all the AppleTalk code has been merged to main now, but disabled by default in the build system. This should allow us to tackle specific improvement areas one by one from now on. I started creating GitHub issue tickets for the known tasks. Please don't hesitate to add more based on your investigations.

Note that I renamed the with-ddp build system option to with-appletalk, since it is arguably much broader that just DDP.

 

[robin-fo]

The PR that brings back all the AppleTalk code

Wow, that was quick!

 

[slipperygrey]

Wow, that was quick!

There is still a lot of work left to do, mind you! This was the easy part, I anticipate...

In all honesty, we have been doing a lot of preparatory work over the last several months, in particular dgsga's hard work backporting the Meson build system to 2.x, plenty of modernization and removal of obsoleted code. This allowed me to practically lift just the relevant source files and build scripts, drop them right into 3.x and have it mostly work.

If you want to keep up with the progress, check out the "appletalk" label in the issue tracker.

 

[Tashtari]

These are really exciting developments! Not sure if I can do anything of value but cheer from the sidelines, but I'm happy to do that, at least.

 

[NJRoadfan]

A good chunk of the work getting the other AppleTalk daemons running has been done as of today. Right now the main branch is a weird hybrid of all AppleTalk only services, except for AFP file sharing, which is still DSI/TCP only.

 

[slipperygrey]

A good chunk of the work getting the other AppleTalk daemons running has been done as of today. Right now the main branch is a weird hybrid of all AppleTalk only services, except for AFP file sharing, which is still DSI/TCP only.

Awesome work so far! This is shaping up nicely. I'll keep an eye out for when you flag your PR as ready for review. My only requirement is that the codebase compiles in its default configuration, and that afpd can start up.

 

[NJRoadfan]

We are one pending PR from being able to enable AppleTalk on the main branch. Most of the low hanging fruit is done. The hard part will be taming the beast that is the afpd service. Quite a lot of code was shuffled around after AppleTalk support was removed.

 

[slipperygrey]

Thanks to NJRoadfan's efforts, AppleTalk is now once more enabled in the main branch, almost 13 years after the previous maintainers started removing it!

I just pushed another changeset that brings back all of the AppleTalk init scripts, so the low hanging fruit have all truly been picked at this point.

 

[NJRoadfan]

AFP reply from localhost:548 via IPv6
Flags: 1  Cmd: 3  ID: 57005
Reply: DSIGetStatus
Request ID: 57005
Machine type: Netatalk4.0.0dev
AFP versions: AFPVersion 1.1,AFPVersion 2.0,AFPVersion 2.1,AFP2.2,AFPX03,AFP3.1,AFP3.2,AFP3.3,AFP3.4
UAMs: DHX2,DHCAST128,2-Way Randnum exchange,Randnum exchange
Volume Icon & Mask: Yes
Flags:
    SupportsCopyFile
    SupportsServerMessages
    SupportsServerSignature
    SupportsTCP/IP
    SupportsSrvrNotifications
    SupportsOpenDirectory
    SupportsUTF8Servername
    SupportsUUIDs
    SupportsExtSleep
    SupportsSuperClient
Server name: debtest
Signature:
09 a7 6b 03 d6 ef ca 68 ca e1 a3 88 37 bf 7f 45  ..k....h....7..E

Network address: 192.168.56.104 (IPv4 address)
Network address: 1.8 (ddp address)
UTF8 Servername: debtest

 

[slipperygrey]

What NJRoadfan is saying in an indirect way here is that we're now looking for alpha testers of Netatalk 4.0.0.

It's a one-way migration path from Netatalk 2.x, so please upgrade only on AFP volumes that have backups, or create fresh volumes for testing.

4.0.0 has a handful of new dependencies, not related to AppleTalk. Notably, Nettle and the Unicode DB. See this pre-release build appendix for concrete packages and commands: https://netatalk.io/4.0/htmldocs/compile ... To get started, clone the latest "main" branch in the Netatalk git repo, and build it with Meson!

Netatalk has to be built with the "-Dwith-appletalk=true" and then you have to put "appletalk = yes" in afp.conf to enable the transport layer. As with 2.x, your OS has to supply appletalk (DDP) networking in the kernel for this to work.

What we want to ascertain right now is twofold: Does AFP-over-AppleTalk work in your environment and workflow. And, is there no performance degradation (file listing, transfer, deletion, etc.)

Looking forward to hearing your feedback!

 

[NJRoadfan]

One thing I haven't extensively tested is the conversion of Apple Double data from the old .AppleDouble folder format to native extended attributes and "dot" files (ex: ._MyFile). So..... back up your shares before trying this! The code should work fine as it wasn't really touched, but always use caution regardless. So far I've tested the following clients:

AppleShare Workstation 1.1 (AFP1.1)
Apple IIgs System 6.0.1 (AFP2.0)
Apple IIgs System 6.0.1 using AFPBridge to connect via TCP/IP (AFP2.0)
The ProDOS 8 2.0.3 client bundled with the above (AFP2.0)
System 7.1 with the stock ASC 3.5 (AFP2.1)
MacOS 8.1 with the stock ASC 3.7.4 via both AppleTalk and TCP/IP (AFP2.2)
macOS 10.15 via TCP/IP (AFP3.4)

Eventually I'll get an XP/2003 image spun up and try Services For Macintosh and PCMacLAN.

Most of this was via emulated machines, so real hardware testing is always appreciated. Get out your rare clients like an Apple IIe Workstation card (@shirsch), or AppleShare for MS-DOS and break this thing! Modern TCP/IP client testing is appreciated as well as I'm concerned about any possible regressions.

 

[Mk.558]

Eventually I'll get an XP/2003 image spun up and try Services For Macintosh and PCMacLAN.

I haven't been able to get 2000 Server with SFM working with AppleTalk over a VirtualBox VM. It only would do AFP over TCP.

PCMacLAN I think should be OK. It's decent. Sluggish under a XP VM instance, and seems to have slight hiccups with AFP over TCP connections, but nothing that breaks it.

 

[NJRoadfan]

PCMacLAN supports native AppleTalk (it includes the transport layer). Microsoft included a native AppleTalk stack in Windows 2000 and Server 2003, but dropped it from XP, hence the need for 3rd party software.

 

[Mk.558]

ya, but I had no luck using Windows 2000 Server with SFM using AppleTalk with a VirtualBox VM. On a real machine it worked perfectly fine -- very well actually -- but IDK what's up with the VM. (It's set to Bridged Adapter)

 

[treellama]

One thing I haven't extensively tested is the conversion of Apple Double data from the old .AppleDouble folder format to native extended attributes and "dot" files (ex: ._MyFile). So..... back up your shares before trying this! The code should work fine as it wasn't really touched, but always use caution regardless. So far I've tested the following clients:

Native extended attributes are the best, easiest way to lose data when you back up files. Will there still be a way to disable this in 4.0?

 

[slipperygrey]

Native extended attributes are the best, easiest way to lose data when you back up files. Will there still be a way to disable this in 4.0?

There is a way, unchanged from netatalk 3.2: "appledouble = v2"

 

[treellama]

Perfect, thank you!

 

[slipperygrey]

You’re welcome! Naturally, a part of this alpha testing process is to ascertain in the field that we didn’t break things like this.

 

[NJRoadfan]

To maintain 100% netatalk 2.x behavior with 3.x and newer, you need to setup the share with the following options.

appledouble = v2
ea = ad

The first option saves resource fork data and classic attributes such as FinderData to a file to the .AppleDouble directory. The second option saves MacOS X and newer's native extended attributes to files in the .AppleDouble directory as opposed to saving them as extended attributes appended to the AppleDouble file.

During the process of testing auto-conversion of Apple Double v2 shares to native EAs and ._files, I found a pretty serious bug in the conversion routines. Any native MacOS X extended attributes stored in the .AppleDouble directory are NOT converted. At least with 2.x shares, the files will remain orphaned in the .AppleDouble directory. How this went undetected is pretty surprising.

 

[slipperygrey]

It is wild that such a bug was not discovered for 13 years. By orphaned, you're saying that Netatalk won't read it again, or is it still read and processed on every access?

At a related note, Netatalk has an option for automatically converting ADv2 metadata to EA metadata, called "convert appledouble". It has been enabled by default since 3.0, but perhaps it is worth discussing whether it should be turned off by default in 4.0. I expect to see a larger number of people migrating their 2.x installations to 4.0, and the conversion might not be what everyone wants. Food for thought.