Netatalk 4.0 - Future-proofing Apple File Sharing

[slipperygrey]

Every now and then, I play around with Netatalk's source code, especially with my attempt to integrate userspace AppleTalk communication which worked with version 2, but requiring some hacks. These hacks were all related to Netatalks frequent use of fork() to generate child processes. While working around problems related to this, I got the impression that forks are somehow regarded as obsolete on macOS. Netatalk even needs some sort of hack to properly run on the latest versions of macOS (OBJC_DISABLE_INITIALIZE_FORK_SAFETY).

Have there ever been considerations to replace forks with threads or other approaches?

I was thinking about this forking situation again recently. Pardon me if this discussed this in detail in a different thread and I forgot about it, but I would like to know more about how afpd's process forking is making your effort in writing a userspace AppleTalk stack more difficult.

Using fork() has some benefits. The ability to spawn one afpd process for each connected user is arguably elegant, compared to juggling threads, mutex and so on in C code in a safe manner. If we have multi-thread C experts in the audience, please correct me.

If I don't misremember, you shared a private repo with me last year that had your modified netatalk code, but I cannot find it right now.

 

[NJRoadfan]

The use of fork() is not obsolete on macOS. The issue is Objective-C, which Netatalk doesn't use: https://sealiesoftware.com/blog/archive/2017/6/5/Objective-C_and_fork_in_macOS_1013.html

 

[slipperygrey]

The problem is that we link with several of Apple's ObjC libraries on macOS, which do have this policy restriction.

 

[robin-fo]

how afpd's process forking is making your effort in writing a userspace AppleTalk stack more difficult

The problem has to do with the AF_APPLETALK sockets which are used to communicate between e.g. afpd and the kernel. In my approach, I replace these with AF_UNIX sockets which works quite well and only requires minor changes to the Netatalk code. On the other end, a userspace program supplies the lower part of the AppleTalk stack (the functionality otherwise implemented in the kernel and atalkd).

Now when the application forks, the AF_UNIX socket gets duplicated which is the cause of two problems: 1) This confuses my userspace stack and could lead to inconsistency with data transmission. 2) When you close one of the two sockets, the stack doesn't know if the socket has beed duplicated and therefore closes the corresponding DDP socket not knowing another process still needs it.

Perhaps you could find a workaround for both scenarios, but there is still that objc problem mentioned above and the severe debug difficulties on the Mac related to forks.

 

[slipperygrey]

What tooling to you use for debugging on macOS? Even on Linux, following forking processes in gdb is hit and miss in my experience. There might be a trick that I'm missing.

Are you not able to use the "OBJC_DISABLE_INITIALIZE_FORK_SAFETY=YES" env variable workaround when developing? The article mentions other methods that we could attempt for a more persistent solution.

FWIW, Apple's ObjC fork policy effectively breaks Python and Ruby as well. For instance the popular requests library in Python, or the entirety of Ruby on Rails. It's a widespread problem...

 

[robin-fo]

What tooling to you use for debugging on macOS? Even on Linux, following forking processes in gdb is hit and miss in my experience. There might be a trick that I'm missing.

I use lldb on Apple Silicon.

 

[robin-fo]

Are you not able to use the "OBJC_DISABLE_INITIALIZE_FORK_SAFETY=YES" env variable workaround when developing? The article mentions other methods that we could attempt for a more persistent solution.

I can (need to) set this, but I think the objc issue is not the reason behind the debugging problems.

 

[slipperygrey]

I can (need to) set this, but I think the objc issue is not the reason behind the debugging problems.

While I don't think this will make a big difference for your issues, this inspired me to apply the other workaround supplied by Apple: To instruct the linker to inject a section in the compiled binary with the contents "__DATA,__objc_fork_ok".

I have implemented and tested it in the PR below. It seems to work correctly and removes the need to set the "OBJC_DISABLE_INITIALIZE_FORK_SAFETY" env variable. The code paths behind the scenes is probably exactly the same though.

For now I'm injecting the section into afpd, cnid_dbd, cnid_metad, and netatalk. In my testing it only seems like the two first are mandatory, but the latter two also contain forking logic. It might be the case that they don't presently link with any ObjC libraries, hence no assert in Apple code is triggered.

On macOS, enable binary section workaround for ObjC fork policy by rdmark · Pull Request #1789 · Netatalk/netatalk

Rather than setting the OBJC_DISABLE_INITIALIZE_FORK_SAFETY env variable, use the linker to inject a section with __DATA,__objc_fork_ok into the resulting macOS binary. This allows for a more persi...

github.com

 

[NJRoadfan]

Here is a response direct from DTS regarding the whole thing: https://developer.apple.com/forums/thread/737464?answerId=764686022#764686022

Basically, if you aren't using Cocoa and stick with POSIX calls, you should be fine. What libraries are being linked by afpd?

With regards to @robin-fo's socket issue, afpd creates a listening socket in main.c, and then creates an ATP response socket in asp_getsess.c in asp_getsession() when it forks.

 

[Mk.558]

@Mk.558 I had the inspiration to fiddle with the webmin module this morning. Can you please give me feedback on this approach to a tabbed interface?

I like it. I'd move Server Status to the front default page (idk...) global config & options to another tab, otherwise, I have no complaint. Someone else might have a better idea though.

...all of the Perl code living in a monolithic index.cgi...

I'm looking nervously at that 633,556 characters in v.4.0 of CMN on just index.html...and it's going to get longer...

 

[slipperygrey]

Here is a response direct from DTS regarding the whole thing: https://developer.apple.com/forums/thread/737464?answerId=764686022#764686022

Basically, if you aren't using Cocoa and stick with POSIX calls, you should be fine. What libraries are being linked by afpd?

With regards to @robin-fo's socket issue, afpd creates a listening socket in main.c, and then creates an ATP response socket in asp_getsess.c in asp_getsession() when it forks.

According to otool, the macOS system libraries we link with are: libSystem, libiconv and LDAP.framework.

otool -L libatalk.19.dylib libatalk.19.dylib: @rpath/libatalk.19.dylib (compatibility version 19.0.0, current version 19.0.0) /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1351.0.0) /opt/homebrew/opt/mysql/lib/libmysqlclient.24.dylib (compatibility version 24.0.0, current version 24.0.0) /System/Library/Frameworks/LDAP.framework/Versions/A/LDAP (compatibility version 1.0.0, current version 2.4.0) /usr/lib/libiconv.2.dylib (compatibility version 7.0.0, current version 7.0.0)

No difference between Sonoma and Sequoia. And, no surprises when inspecting all the other daemon binaries.

So, this got me thinking, can't we link with openldap instead of the Apple Framework? And well, we can:

brew install openldap meson setup build -Dwith-ldap-path=/opt/homebrew/opt/openldap meson compile -C build otool -L build/libatalk/libatalk.19.dylib build/libatalk/libatalk.19.dylib: @rpath/libatalk.19.dylib (compatibility version 19.0.0, current version 19.0.0) /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1351.0.0) /opt/homebrew/opt/mysql/lib/libmysqlclient.24.dylib (compatibility version 24.0.0, current version 24.0.0) /opt/homebrew/opt/openldap/lib/libldap.2.dylib (compatibility version 3.0.0, current version 3.200.0) /usr/lib/libiconv.2.dylib (compatibility version 7.0.0, current version 7.0.0)

 

[slipperygrey]

I like it. I'd move Server Status to the front default page (idk...) global config & options to another tab, otherwise, I have no complaint. Someone else might have a better idea though.

Thanks for your feedback!

Do you mean moving the "Show Server Status" link to the default tab (File Sharing Services), and keeping the rest under Global Settings, correct?

I can see how that would make sense, in that a user would want to check the AFP server's response on the same page that they control said server.

The only drawback would be a loss of symmetry of having the four icon links in a single row.

 

[NJRoadfan]

Linking OpenLDAP libraries appears to be the correct thing to do. Apple has "LDAP.framework" labelled as "do not use" and clearly that is the case. Folks are having App Store submissions rejected over this. Officially, LDAP appears to be handled by the Open Directory framework. Seems like a common bug in build environment detection of LDAP libraries on mac OS systems.

The rest of the libraries should be C only. Maybe it was a good thing that macOS had that pesky fork message?

 

[slipperygrey]

Look at that, Apple’s docs corroborate this assertion. (Table A-1)

OS X Frameworks

Introduces OS X and its technologies.

developer.apple.com

Good thing we had this conversation! Let me tweak the build system to favor openldap on Darwin.

 

[slipperygrey]

This seems to do the trick. If the Homebrew openldap library is present, it will be used by default now.

meson: Favor openldap when building on macOS by rdmark · Pull Request #1792 · Netatalk/netatalk

On macOS when doing find_library('ldap') Meson will find LDAP.Framework libraries which are deprecated by Apple. Ref. https://developer.apple.com/library/archive/documentation/MacOSX/Concep...

github.com

Eventually, I want to remove the "OBJC_DISABLE_INITIALIZE_FORK_SAFETY=YES" workaround from the netatalkd script, but not in the stable 4.0 release series. There may be older macOS versions where we still pull in other non-fork-safe Apple frameworks somehow...