Jump to content

Serial-based remote code execution idea for enabling modern software on classic Macs


Recommended Posts

Hey everyone, 

 

I'm a long time programmer and have recently become interested in classic Macs. My family had a Mac Plus when I was a kid and I've been recently thinking about how interesting it would be to make a machine like that "useful" again. The thought has been keeping me up at night recently, and now I've come up with what I think might be a solution.

 

Firstly, this approach does not allow old software to become internet connected or anything like that. It is purely aimed at creating new software or allowing the creation of new software.

 

My idea is to create a modern piece of software that allows programmers on any old machine to hand off work loads off to a modern machine over the serial port and get a response back that they can then interact with. This software will provide a standard interface for serial-connected devices to provide code to execute.

 

For my first approach, I am going to have workloads all be in modern nodejs. I like nodejs because it has a vast ecosystem of easy to use packages via npm, and if you are using an old computer, should be "fast enough" to do whatever it is you are trying to do. 

 

Now, I can say all that and it's easy to do so without having written any code. I decided to write some code and I have a loose first attempt with a more technical README up at: https://github.com/CamHenlin/coprocessor.js I've proven to myself that the strategy can basically work via unit tests.

 

The next step for me is to get serial working with an old Macintosh emulator (I don't have a real Macintosh yet :( - so expensive on eBay!), then begin work on a C library to wrap the serial communications on the classic Macintosh end. I will publish this library on GitHub as well with very easy instructions.

 

Ultimately, I'd like to take this far enough to get a Twitter client working on whatever 68k Mac I ultimately get my hands on. Hopefully others might find the idea useful enough that they could develop interesting software for old Macs as well.

 

What do others here think? Has anyone tried an approach like this? Does anyone have any additional thoughts, questions, or anything that I'm not seeing? I'm interested in hearing your feedback. Thanks!

Link to post
Share on other sites

I'm not sure how useful this is, but the idea just sounds like an awesome hack! :)

 

It's just like the C64 disk drives that essentially used the same processor as the C64 itself. People built programs back in the 1980s that used the disk drive as a copprocessor!

 

Also see this current thread

 

And last but not least. Considering there is a modern SSH client available for 68040 machines, it should be possible to build a Twitter client that runs on this directly. But Twitter on a 68000 would be even cooler in my view!

Link to post
Share on other sites
  • 68kMLA Supporter

Coming from the Acorn things, I'm a big fan of coprocessors and being able to build modern ones ;-).

 

I'd like to suggest, though, that perhaps the serial port is not the best way to go about this; it will work, but it will limit your options later, both because (modulo certain corner cases) you're using up a whole port just for your copro, and partly because getting the serial ports to go fast is a non-trivial exercise.

 

What I was thinking of doing to play with this kind of idea was to integrate with RaSCSI to talk over the SCSI port.  This essentially gives you packetised communication between the host and the coprocessor at a much higher speed than you could achieve through a serial port, you can daisy-chain devices, and it lets you mount your coprocessor internally.  Lots of work has been going on to make RaSCSI mac-friendly and English-ify the codebase, and that would strike me as perhaps a better place to be.

 

Harder to emulate, mind...

Link to post
Share on other sites
  • 68kMLA Supporter
Posted (edited)

This is a cool idea.  +1 to doing it with SCSI!  Then you could do things like import a screen-size B&W bitmap (22 KB) computed by a modern machine to a Mac Plus in probably a couple seconds?  The possibilities from there are endless ...

Edited by Crutch
Link to post
Share on other sites
22 hours ago, cheesestraws said:

Coming from the Acorn things, I'm a big fan of coprocessors and being able to build modern ones ;-).

 

I'd like to suggest, though, that perhaps the serial port is not the best way to go about this; it will work, but it will limit your options later, both because (modulo certain corner cases) you're using up a whole port just for your copro, and partly because getting the serial ports to go fast is a non-trivial exercise.

 

What I was thinking of doing to play with this kind of idea was to integrate with RaSCSI to talk over the SCSI port.  This essentially gives you packetised communication between the host and the coprocessor at a much higher speed than you could achieve through a serial port, you can daisy-chain devices, and it lets you mount your coprocessor internally.  Lots of work has been going on to make RaSCSI mac-friendly and English-ify the codebase, and that would strike me as perhaps a better place to be.

 

Harder to emulate, mind...

 

This was my thought I have been expressing for years, but not really any time to do. 

 

The idea is to build an ARM-based coprocessor PDS card, with all the proper bus interface logic so the 68K can speak to the ARM, and vice-versa. This would allow people to develop apps on the mac, but then allow the ARM to do the heavy lifting for math and memory intensive operations, such as running Chromium engine for a browser, but there is a driver on the Mac side for shared memory space, as well as shared resources. so it would still use the Mac's network stack, and other things. but use the ARM simply for coprocessing only. the Bus logic/ARM would ideally be able to work in bus mastering mode, so the ARM could DMA the data requests back into the Mac's RAM, or... the card could emulate a slice of RAM on the 68K's memory bus at the specified peripheral addressing. 

 

This would be the fastest way to hand off data between the ARM and the 68K. Serial would be a bad idea. and SCSI is a much better option. PDS is still the best option, but SCSI is a much more universal option I suppose, albeit not as fast. 

 

You would still have to write a mac-side driver either way. 

 

This could enable things such as Electron apps to run in a Classic macintosh environment, which would be cool. But the next limitation is going to be the video hardware subsystem. So youll have to work around that. (no youtube videos, sorry). otherwise you would need to figure out a way to take over framebuffer and then the ARM could overlay itself. but then the Mac is just a terminal at this point. 

Edited by techknight
Link to post
Share on other sites
  • 68kMLA Supporter
20 hours ago, techknight said:

PDS is still the best option, but SCSI is a much more universal option I suppose, albeit not as fast. 

 

Yup, PDS would be much faster and the "ideal" answer; the main reason I tend to suggest SCSI in these conversations is purely that the number of people who have the skills to build stuff to plug into SCSI is much higher than the number of people who could build a PDS card from scratch, which increases the chance that this might actually happen :-).  But this is a purely pragmatic concern; in principle, I absolutely agree with you

Link to post
Share on other sites
20 hours ago, cheesestraws said:

 

Yup, PDS would be much faster and the "ideal" answer; the main reason I tend to suggest SCSI in these conversations is purely that the number of people who have the skills to build stuff to plug into SCSI is much higher than the number of people who could build a PDS card from scratch, which increases the chance that this might actually happen :-).  But this is a purely pragmatic concern; in principle, I absolutely agree with you

Yeah thats why i mentioned SCSI would be the most "compatible" option. because then you can use lowly little ol powerbooks. 

 

Edit: You know, I just saw another thread that gave me an idea:

 

Edited by techknight
Link to post
Share on other sites

@techknight pointed me here from a thread I had coincidentally started. :)  I don't know if @camh was aware of A/ROSE, few people were.  It does exactly as he described and is already documented and supported on the 68k (for the literally 3 or maybe 4 cards that actually utilized it.)  It was an SoC that handled the heavy duty work for more advanced cards, then simply passed back the results, freeing up the CPU to do other things.

 

I was positing in my thread that perhaps it was time to revisit A/ROSE.  With FPGAs and stuff, presumably someone could make an affordable A/ROSE clone.  A/ROSE is already documented and everything, so no additional software or APIs would be required to implement it.  It could open the doors for faster SCSI cards, USB cards, multi-port ethernet cards, combo interface cards, etc.

 

Link to post
Share on other sites
On 3/9/2021 at 11:17 PM, camh said:

My idea is to create a modern piece of software that allows programmers on any old machine to hand off work loads off to a modern machine over the serial port and get a response back that they can then interact with. This software will provide a standard interface for serial-connected devices to provide code to execute.

You just described System 7 Program Linking. Check out Inside Macintosh volume VI.

Link to post
Share on other sites
  • 68kMLA Supporter
20 hours ago, Dog Cow said:

You just described System 7 Program Linking. Check out Inside Macintosh volume VI.

 

OP just described RPC.  There are plenty of other RPC mechanisms than program linking, even over AppleTalk.  Program Linking is far better suited to using for peer-to-peer dynamic application-to-application messaging than coprocessor communication, anyway.  I suspect something like the Tube coprocessor interface would be a better inspiration.

Link to post
Share on other sites
Posted (edited)

Hey everyone, I'm glad to check back and see that there is some interest in the project! I love all the feedback, thoughts, suggestions, etc. I'm going to forge ahead with this path for now but might consider some other stuff once I get it working and find its limitations.

 

I've been doing a bit more tinkering on the 68k Macintosh side and have a small C program running in the emulator capable of dumping out data that can be read by the nodejs library that I posted: https://github.com/CamHenlin/retro68-coprocessorjs-test. It can't handle responses yet but will as soon as I am able - this should at least give people a better idea of how this will work. All of the interesting code is in this file for now: https://github.com/CamHenlin/retro68-coprocessorjs-test/blob/main/hello.c - and overall the project structure was taken from: http://www.toughdev.com/content/2018/12/developing-68k-mac-apps-with-codelite-ide-retro68-and-pce-macplus-emulator/. Honestly struggling with the provided VM having issues due to being way out of date, and want to set up the dev workflow on my MacBook without the VM before proceeding much further

 

So now we've got:

https://github.com/CamHenlin/coprocessor.js - capable of reading serial "messages" in various forms, executing on them, sending responses back

https://github.com/CamHenlin/retro68-coprocessorjs-test - capable of sending out serial messages but not receiving the responses yet. The example serial messages include a full nodejs application, function execution commands, and eval commands

 

 

What's next?

- Fix my dev workflow

- Make the test app handle responses

- Handle the gobs of TODO comments

- Wrap things in to a more consumable library

- Try to use the library in a more interesting GUI-based application

 

Edited by camh
Link to post
Share on other sites
21 hours ago, camh said:

I'm going to forge ahead with this path for now.

 

 

 

Well.... to each their own I suppose. Good Luck! (you're going to need it. serial isnt fast by any means).

 

I suppose it could be a proof of concept. But not sure how much of it would be portable to different and faster standards. 

Edited by techknight
Link to post
Share on other sites
Posted (edited)
21 hours ago, techknight said:

 

Well.... to each their own I suppose. Good Luck! (you're going to need it. serial isnt fast by any means).

 

I suppose it could be a proof of concept. But not sure how much of it would be portable to different and faster standards. 

 

I believe 68000-based Macs should be able to achieve 19200bps over serial if LocalTalk is not being used, meaning about 2.4kbytes/s max speed - that's fast for raw text and that's also plenty for interacting with nodejs scripts and dealing with the results. I'll provide some better examples of writing some applications in this fashion soon (once I get it fully working ! :)),) and maybe you'll agree with the approach once you've seen it

Edited by camh
Link to post
Share on other sites
On 3/10/2021 at 8:47 PM, camh said:

 

I believe 68000-based Macs should be able to achieve 19200bps over serial if LocalTalk is not being used, meaning about 2.4kbytes/s max speed - that's fast for raw text and that's also plenty for interacting with nodejs scripts and dealing with the results. I'll provide some better examples of writing some applications in this fashion soon (once I get it fully working ! :)),) and maybe you'll agree with the approach once you've seen it

 

yea, I suppose seeing is believing. I suppose if you use some type of compression method you could get something decent, but decompressing will take CPU cycles on the Mac side. so no free lunch. 

 

I am thinking along the lines of a decent modern browser, but the engine is run over a coprocessor. 

Link to post
Share on other sites
On 3/11/2021 at 11:10 PM, techknight said:

 

yea, I suppose seeing is believing. I suppose if you use some type of compression method you could get something decent, but decompressing will take CPU cycles on the Mac side. so no free lunch. 

 

I am thinking along the lines of a decent modern browser, but the engine is run over a coprocessor. 

Some fast decompressor like lz4 or lzo could reduce the CPU cycles needed while allowing faster transfers.

Edited by uyjulian
Link to post
Share on other sites

I have a short but positive update!

 

As I stated in my last update, I needed to fix my dev workflow before proceeding much further. After much learning, I think I've accomplished that. I'm definitely going to write a blog post on my personal setup later because there are a lot of gotchas during setup.

 

Ultimately I now have an older computer with latest ubuntu running codelite mostly based on http://www.toughdev.com/content/2018/12/developing-68k-mac-apps-with-codelite-ide-retro68-and-pce-macplus-emulator/ with tty0tty providing a virtual serial port to the test app in pce/macplus as well as the nodejs coprocessorjs service. That means I now have a setup where I am testing something close to real serial communications on the 68k Mac and nodejs side.

 

The even better news is that after getting all of that set up, I was able to relatively quickly get the Retro68 test app code at https://github.com/CamHenlin/retro68-coprocessorjs-test and the nodejs code at https://github.com/CamHenlin/coprocessor.js updated and working with everything being treated as actual serial communication. I have the Retro68 test sending data over the virtual serial port and the nodejs service is responding to it.

 

Next steps are now:

- Get the retro68 test app reading data back from the nodejs service

- Handle TODO comments, mostly around cleanup

- Create more consumable library that can be used in other apps

- Create a more interesting app with the library

Link to post
Share on other sites
On 3/15/2021 at 6:31 AM, camh said:

I have a short but positive update!

 

As I stated in my last update, I needed to fix my dev workflow before proceeding much further. After much learning, I think I've accomplished that. I'm definitely going to write a blog post on my personal setup later because there are a lot of gotchas during setup.

 

Ultimately I now have an older computer with latest ubuntu running codelite mostly based on http://www.toughdev.com/content/2018/12/developing-68k-mac-apps-with-codelite-ide-retro68-and-pce-macplus-emulator/ with tty0tty providing a virtual serial port to the test app in pce/macplus as well as the nodejs coprocessorjs service. That means I now have a setup where I am testing something close to real serial communications on the 68k Mac and nodejs side.

 

The even better news is that after getting all of that set up, I was able to relatively quickly get the Retro68 test app code at https://github.com/CamHenlin/retro68-coprocessorjs-test and the nodejs code at https://github.com/CamHenlin/coprocessor.js updated and working with everything being treated as actual serial communication. I have the Retro68 test sending data over the virtual serial port and the nodejs service is responding to it.

 

Next steps are now:

- Get the retro68 test app reading data back from the nodejs service

- Handle TODO comments, mostly around cleanup

- Create more consumable library that can be used in other apps

- Create a more interesting app with the library

 

Once you make this work, you should consider doing a hackaday.io blog on it. My project got so much interest, it was on the front page of hackaday yesterday. 

 

Something to think about :-)

Link to post
Share on other sites

Had a great evening hacking on the project and just posted lasted code updates on github. I'm happy to report that full bi-directional communication is working between the C app on 68k Mac and the nodejs service on the host machine via the virtual serial port and have handled the majority of the TODO items in the codebase. Here is a screenshot of the emulator running the test application and the nodejs coprocessor.js host application running on the other window. In this test app, the 68k Mac app is sending the coprocesor.js service on the host a full nodejs app, then requesting that a function be run on that nodejs app and getting the results back into the 68k mac app, all via virtual serial cable. This should pretty easily apply to a real serial cable as well. 

 

1674167401_Screenshotfrom2021-03-1700-05-52.thumb.png.d9009c62b1359089636e5b45af5ef847.png

 

The nodejs app being sent out here is silly and just installing the lodash library, and running a single function on it, but it would be trivial to expand this test app to run any nodejs script or package on the host, including internet-connected ones, and get the data back on the 68k Mac side. 

 

So next steps:

- handle the few remaining TODO items. some TODO items are future research items and I probably won't bother addressing them for now

- adding a step here of making the test app more interesting with a more interesting nodejs script, should be pretty simple

- modularize the Retro68 C code into an easily consumable library

- create a more interesting app that uses the library

- document the process so others can write apps against the coprocessor.js host using the library if they're interested

Link to post
Share on other sites
Posted (edited)

I've got another update to share! I think this whole idea looks more and more realistic every time I sit down with it. 

 

I've spent the last few days knocking out some of the "next steps" from my last post. I've got all of the important TODO items figured out in my C test code and it does everything that I feel is needed to turn it in to a library: it has some basic function that will be exposed that could be used in other programs or libraries, is pretty fast, doesn't leak memory, handles "big" responses from the nodejs server at the other end of the serial port.

 

The basic functions that will go into a library include: one for loading nodejs programs over the serial port, one function for calling functions in the nodejs program over the serial port back and getting a response, one for evaluating raw string code within the nodejs app on the serial port and getting a response. The current iteration of the code allows chunks of data in a function or eval response to reach up to 100kb, beyond that a programmer will have to manually chunk data. That's really a lot of data and should allow for some interesting stuff.

 

In addition to that, I've worked out a simple shell script for packaging complete nodejs applications in to a C application to send over the wire, and used this strategy to create a slightly more interesting Retro 68 C test application. The more interesting test application now includes a nodejs application that sets up puppeteer (headless chrome) on the remote system, and uses puppeteer to grab the text contents of this thread, then dumps that text back across the serial port. I think this is pretty neat. This code is up at https://github.com/CamHenlin/retro68-coprocessorjs-test

At this point, I think someone could use the Retro68 test application to create just about any terminal-based application they want, with all of the heavy lifting done in modern JS on the other side of a serial port, as long as the Retro68 console suits their needs. Writing Internet-conected applications is now easy, without mucking around with old TCP libraries or expensive hardware. 

 

1995813811_Screenshotfrom2021-03-1923-16-44.png.090223a53efaad12fc02146c14d8209e.png

 

 

Now next steps are: (maybe not fully in order)
- modularize the Retro68 C test code at https://github.com/CamHenlin/retro68-coprocessorjs-test into an easily consumable library (basic functions mentioned above will be available)
- create a more interesting app that uses the library (simple twitter client first?) probably need to tinker around with this quite a bit on the Mac programming side
- create a skeleton project structure + blog post outlining dev workflow, how others could create coprocessorjs apps using the library. Create simple VM here maybe?
- create another blog post about setting up physical hardware, between whatever 68k Mac I end up buying and a raspberry pi. Create shareable SD card image for a raspberry pi here.
- create some more apps or address library issues as they come up. basically any library on https://www.npmjs.com/ could be used to back a Macintosh app using this tool

 

 

 

Edited by camh
Link to post
Share on other sites
  • 68kMLA Supporter
On 3/19/2021 at 10:30 AM, camh said:

In addition to that, I've worked out a simple shell script for packaging complete nodejs applications in to a C application to send over the wire, and used this strategy to create a slightly more interesting Retro 68 C test application

 

Am I understanding this right?  The Mac side can essentially send a nodejs application over to the coprocessor and have it run it, so you only have to distribute one binary (not one for the copro and one for the host Mac)?  If so, that's very neat, well done. :-).

Link to post
Share on other sites
22 hours ago, cheesestraws said:

 

Am I understanding this right?  The Mac side can essentially send a nodejs application over to the coprocessor and have it run it, so you only have to distribute one binary (not one for the copro and one for the host Mac)?  If so, that's very neat, well done. :-).


Yes! Right!

 

https://github.com/CamHenlin/retro68-coprocessorjs-test/blob/main/output_js.h Is an entire nodejs app included in the C build generated by https://github.com/CamHenlin/retro68-coprocessorjs-test/blob/main/compile_js.sh. The raw JS is here: https://github.com/CamHenlin/retro68-coprocessorjs-test/tree/main/JS

 

and you can see how that gets used in the C code here:

 

https://github.com/CamHenlin/retro68-coprocessorjs-test/blob/d7b6b9c166d9cab0baebc18648c22fd9162bb440/hello.c#L566 Sends the nodejs program over the serial port

 

https://github.com/CamHenlin/retro68-coprocessorjs-test/blob/d7b6b9c166d9cab0baebc18648c22fd9162bb440/hello.c#L574 Reads the response back

 

It’s that easy! The host software on the serial-connected machine takes care of the rest. The programmer on the classic Mac side doesn’t have to worry about much. 

Link to post
Share on other sites

This is pretty neat!

 

It seems to me that this could open up the possibility of a web browser that can offload most of the JS/CSS rendering and security protocols to a reasonably fast co-processor and send the results back to the Mac, which only needs enough oomph to render the HTML and push the pixels around the display.  It probably couldn't do much (it certainly couldn't reasonably handle the full desktop versions of, say, YouTube, Facebook or Twitter due to various bottlenecks in the Mac, though mobile versions of each might be possible, with limitations), but perhaps it would work well enough to make visiting relatively basic sites like 68kMLA feasible.

 

This way, it's not quite like cheating, because the Mac itself is rendering the site, but the heavy lifting is handled by a co-processor, not unlike a spreadsheet or heavy-duty mathematical program offloading heavy computations to an FPU or some other dedicated co-processor).   Plus, it can be more interactive than the rendering proxies out now, because the end result is assembled on the Mac itself, with help from the co-processor to assemble some of the building blocks (my understanding is that most rendering proxies create what amounts to a static screenshot of a site with a link map overlaid onto it).

 

c

Link to post
Share on other sites

Did some code slinging tonight and have some more work to share.

 

Firstly, I cleaned up the code in my Retro68 test application at https://github.com/CamHenlin/retro68-coprocessorjs-test and moved all of the things necessary to use coprocessorjs from the classic Macintosh side in to a separate library for inclusion in other projects.

 

Next, I moved the library file to a discrete repository with a lengthy README explaining how to use the library to build a coprocessorjs application. That new repo is at: https://github.com/CamHenlin/coprocessorjslib. I tried to make the README as clear as possible, let me know if you have any questions about it. I intend to consume this library in a more interesting application soon.

 

So, moving right along with that checklist. Additionally, I was finally able to score what I felt like was a good deal on a Macintosh Classic, so should be able to test this out on real hardware soon! I have what I think is a pretty good strategy worked out for that already, so I'm really excited for it to show up.

 

 

Link to post
Share on other sites

This is really cool - I think the serial port is under-rated for connecting these old machines to newer machines. I know there are some other projects out there using ESP8266 including https://www.tindie.com/stores/theoldnet/ but I'll just add mine to the list and a possible alternative to jsnode - micropython can be flashed on some of these ESP8266 boards, I'm using NodeMCU, and they are quite powerful and have a tiny footprint compared to a larger machine running Ubuntu. You can even expose the Python REPL over the serial port if you wanted to. The Python libraries are not complete but for an embedded subset, it's surprisingly large and feature-rich. My project is VERY simple and not sophisticated, I'm doing really bad timing/buffering stuff just to get a few commands across to Hypercard but I mostly did this as a proof-of-concept to see what is possible with this combination. I also love the idea of creating new applications that depend on a more featured bridge machine.

 

https://github.com/cryptoterminals/maccryptoterminals

 

I didn't totally follow how you got your C code onto the Mac, maybe I missed the documentation, are you compiling outside in CodeLITE and then moving the binary to the emulated Mac? Would I be able to use MPW or ThinkC to build what you have in your Github directly on my Mac Plus?

 

Cool stuff!

Link to post
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

Loading...
×
×
  • Create New...