Netatalk 4.0 - Future-proofing Apple File Sharing

[slipperygrey]

The current Netatalk release series (2.x and 3.x) are finally in a good place where we have a modern and flexible build system, with fully functioning user authentication. I can finally pivot to what I wanted to do in the first place: consolidate the best of 2.x and 3.x into one Mac file sharing suite to rule them all.

I hereby present: The Netatalk 4.0 Roadmap.

The mission statement is: A future-proof file sharing suite for Apple //e to macOS and everything in-between.

In the long run, I want to stop supporting two very different branches and have one branch will all the code modernization of 3.x with the support for old clients that 2.x provides.

At a high level, these are the projects:

AppleTalk​

- Graft the AppleTalk protocol modules back onto libatalk: asp, atp, nbp, ddp
- Bring back the AppleTalk daemons: atalkd, papd, timelord, a2boot
- Bring back the AppleTalk networking tools: aecho, getzones, nbp, pap
- Bring back afpd support for AFP 1.1, 2.0, and 2.1

Modern Tech​

- Write a CNID backend in SQLite (Berkeley DB is abandonware)
- Write a Spotlight indexing and search backend in Elasticsearch (Gnome Tracker has not been working well)
- Use Nettle as crypto backend (eliminate remaining the dependencies on OpenSSL)
- Use GDBus as D-Bus client (dbus-glib is going away)

On top of this, I want to remove the old Autotools build system, shore up all the insecure memory mangement, write unit tests...

Needless to say, this will be a massive undertaking. I am looking for collaborators to make this happen. If you or someone you know has a knack for C (or is willing to learn) I can offer a job that is unpaid but comes with the glory of keeping the dream of Apple file sharing alive for the next generation.

 

[dramirez]

Thanks for your hard work!

 

[robin-fo]

I‘ll follow this with interest..
Additional suggestion: Provide a well-defined DDP socket interface to let libatalk work with userspace AppleTalk stacks.

 

[slipperygrey]

I‘ll follow this with interest..
Additional suggestion: Provide a well-defined DDP socket interface to let libatalk work with userspace AppleTalk stacks.

That's a great idea! Don't hesitate to lean in and... err... design said interface to fit your project's needs precisely.

 

[cheesestraws]

Additional suggestion: Provide a well-defined DDP socket interface to let libatalk work with userspace AppleTalk stacks.

agreed!

That's a great idea! Don't hesitate to lean in and... err... design said interface to fit your project's needs precisely.

also agreed! my efforts down this road are somewhat stalled for lack of mental health, and in some nebulous future when I have the ability to attack nontrivial projects again I will happily fall in with whatever interface you decide suitable.

Thanks for your hard work!

also also agreed! What you're doing here is much needed, @slipperygrey, and it's great work. I am not going to volunteer for anything because at the moment I would mostly be a disappointment, but hopefully things will improve and I can come and join in.

 

[MacKilRoy]

Awesome!!

 

[Mk.558]

Looking forward to this. At the current time I recommend Netatalk for the widest range compatibility and stability.

MacIP gateway inclusion is coming too yes?

 

[slipperygrey]

Looking forward to this. At the current time I recommend Netatalk for the widest range compatibility and stability.

MacIP gateway inclusion is coming too yes?

MacIP is still on my radar. What I want to aim for is an independent macipgw that can easily be linked with netatalk's libatalk. For starters, I'm looking into improving the Debian packaging of netatalk so that you can install the dev headers and shared libraries, lowering the bar to building macipgw.

I reached out to Stefan (the original author) over email a few weeks ago, but he hasn't responded yet. Before potentially putting macipgw under the netatalk umbrella formally, I want to give him the opportunity to weigh in.

 

[Mk.558]

ya I was thinking of that thread on github yesterday evening. Between NJRoadfan, yourself and a few others, I literally cannot think of anybody more qualified to work on such a thing at the moment. But these things are powered by passion, and if the passion isn't there, that's just the way it is.

Other people may not recognize the significance of this kind of work. I value you and your mate's work on this matter.

 

[slipperygrey]

ya I was thinking of that thread on github yesterday evening. Between NJRoadfan, yourself and a few others, I literally cannot think of anybody more qualified to work on such a thing at the moment. But these things are powered by passion, and if the passion isn't there, that's just the way it is.

Other people may not recognize the significance of this kind of work. I value you and your mate's work on this matter.

Thank you for the kind words! It's the thrill of the problem solving that keeps me going. And the response from the community of course. I'm able to keep going for now, but my time and energy is definitely a limited resource.

My collaborator Dave, who was the driving force behind many of the big changes over the last 12 months, had to leave the project for personal reasons recently, so I'm on my own right now. He pulled off some incredible feats of software engineering in a 30 year old codebase to get us where we are now, and I'm hugely indebted to him.

 

[twelvetone12]

Firstly thanks @slipperygrey for your great work on netatalk! I feel I could contribute maybe, but since last year somewhat life happened and I got stuck with all my Mac projects. I'm trying to start again (like getting my localtalk module into Linux) so maybe I could help with smaller things...

 

[slipperygrey]

Firstly thanks @slipperygrey for your great work on netatalk! I feel I could contribute maybe, but since last year somewhat life happened and I got stuck with all my Mac projects. I'm trying to start again (like getting my localtalk module into Linux) so maybe I could help with smaller things...

I hear ya. Are you in a more stable situation now? Last year was one of many life events for me too and I had to effectively drop all my retro hardware tinkering projects and get rid of most of my collection on short notice. Netatalk has been a nice diversion and a way to keep a mental foot in the door with this community!

If nothing else I'd love to get your code reviews. Right now I'm pushing code to main with impunity and may be making obvious mistakes that will bite me later.

You can have a look at the issue tracker. The ticket backlog is well maintained, so if you spot a bug or idea that piques your interest, then don't hesitate to give it a shot! Once the project of porting over the AppleTalk code gets off the ground, there will likely be a bunch of tasks related to that which could be delegated.... Stay tuned!

 

[cheesestraws]

If nothing else I'd love to get your code reviews.

I'm happy to do code review too if helpful, I think that is within my current grasp.

 

[hauke]

You rock!

I hope I can contribute more time than in the recent past...

 

[slipperygrey]

I'm happy to do code review too if helpful, I think that is within my current grasp.

Excellent! So for one you can start watching the GitHub project and lean in when you see a PR pop up. And, I'm going to ping you if I put up a particularly complex and risky one that could really use a second set of eyes...

 

[slipperygrey]

You rock!

I hope I can contribute more time than in the recent past...

Cheers! Thanks for all your guidance in the past.

When you get a chance to try again, I'd love to know if the BerkeleyDB lib dir error on NetBSD went away for you or not.

 

[cheesestraws]

Excellent! So for one you can start watching the GitHub project and lean in when you see a PR pop up. And, I'm going to ping you if I put up a particularly complex and risky one that could really use a second set of eyes...

Am watching! FWIW my github username is the same as it is on here.

 

[cheesestraws]

Excellent! So for one you can start watching the GitHub project and lean in when you see a PR pop up

Well, you put one (or three) in I found I understood so I've weighed in on them. Hopefully as I get more used to the codebase I'll become more useful...

 

[NJRoadfan]

I've been going through the 3.x afpd code base seeing what needs to be restored. Eventually I'll get my notes up, but there are two parts of this so far:

1. Adding back pre-AFP 2.2 compatibility: Thankfully most of this is still in place in the code base. The biggest change is restoring the ProDOS and free space size limiting volume flags from 2.x.

2. Adding back DDP/ATP transport support: This is going to be where most of the work needs to be done. The 2.x code base spawns separate child process types for DDP/ATP or TCP/DSI depending on the connection. In 3.0, most of this was simplified and hard coded for TCP/DSI.

 

[slipperygrey]

Well, you put one (or three) in I found I understood so I've weighed in on them. Hopefully as I get more used to the codebase I'll become more useful...

Cheers, this was helpful!

I've been mostly noodling with the Meson build scripts lately, which consist of domain-specific pseudo-Python that takes a bit of getting used to. With the AppleTalk restoration project we will be heading into proper C coding shortly.