Attempting To Program Something Using LLMs With Zero Knowledge or Skill

[olePigeon]

I get Google Gemini for free at work. So I've been messing around with it. Currently I am attempting to make a tiny database program using THINK C version 6 on System 7.

I attempted to start with Codewarrior, but it was a complete failure. I then tried Symantec C++ and it was a bit better, but it was just constant syntax errors. So now I'm on THINK C v6 and it's going along somewhat smoothly.

I told it to limit its knowledge to a handful of period appropriate books. Even though I tell it to use C and not Pascal, it keeps trying to sneak Pascal in there and breaks the program. I think it gets confused on the code examples it's pulling from where they give both Pascal and C examples for the same thing.

Anyway, it's coming along surprisingly far given I have zero experience. I'm also learning a bit about how C works and the Macintosh Toolbox. I guess it's just sort of a nerdy sandbox. Just playing around.

 

[CC_333]

You may have landed on one of the relatively few genuinely helpful uses I can think of for LLMs.

I'd be lying if I didn't say I wasn't just a tiny bit curious, but I'm a Luddite who looks at this newfangled technology (and the companies behind it) very distrustfully, and I don't want to give them any of my money or information if I can help it.

If I didn't have so many qualms about it, I'd ask it to write me a stream player app for Mac OS 7 ,8 and 9 (plus a Carbonized version that can also run natively on Mac OS X 10.0.x-10.4.x) that's lighter and more purpose-built for streaming than iTunes and Sound Jam MP, easier to use than MpegDec, and can support at least a useful subset of modern security protocols and encryption ciphers so it can connect to encrypted streams. Kind of like a Spotify or Pandora app (I think someone actually has made such a thing for the former, so there's some precedent), but more platform agnostic where all you do is feed it a URL to a stream, and it simply plays it and nothing more.

Or better yet, how about a fully up to date Mozilla-like browser (kind of like Firefox, but for Classic Mac OS) running with at least basic HTML5 support, which has this facility built in?

c

 

[finkmac]

ok

 

[Durosity]

You may have landed on one of the relatively few genuinely helpful uses I can think of for LLMs.

I agree. I’ve always been awful with programming.. just my brain will not do it. But I have been using Claude to build plugins for my Home Automation system Indigo Domotics and it’s done an amazing job. I mean.. it’s a bit of a fight.. I feel like I’m in control of an autistic toddler who’s been given a multi pack of red bull.. but when properly controlled it’s doing great work.. it’s managed to integrate things I’d never thought possible!

 

[cheesestraws]

The problem is that, given a problem, writing the code to solve the problem isn't the hard bit. It's knowing whether it will break or not, and working out how, that's the hard bit.

 

[adespoton]

The problem is that, given a problem, writing the code to solve the problem isn't the hard bit. It's knowing whether it will break or not, and working out how, that's the hard bit.

Modifying code written by an LLM is also a challenge.

I've found one really good use for them: distilling large volumes of text into something actionable. Sometimes I get LLMs to script up something for me as well, but I make sure they do it in a manner where I can visually check all the components and easily identify the stuff it just makes up and fix it -- so it saves me time, but doesn't waste my time.

Actually, I lie: along with the distilling large volumes of text, LLMs are also good at doing a first code review and suggesting things I should fix in my code -- it'll make a bunch of stuff up that I can ignore, but it can also catch some obvious and not so obvious mistakes I've made, allowing me to fix them and learn from them.

 

[olePigeon]

I think it would be very useful for an accomplished programmer who knows what they're doing and has a large repository of code they've written. The LLM can can train off that repository and produce code that the programmer is accustomed to and would find it relatively easy to follow and debug.

Writing from scratch using whatever random stuff the LLM has stolen off the internet ... not so much.

 

[ArmorAlley]

I write SQL for a living (which is sort of code) and while I often find LLM-generated SQL impressive, you really need to know what the right answer is, because otherwise you can't judge how useful or correct the generated SQL is.
I have thought that LLMs might be great for someone who is already well-versed in the Assembly Language used to write software up to System 6, once it can be fed in appropriate texts, code and so on.

 

[adespoton]

I write SQL for a living (which is sort of code) and while I often find LLM-generated SQL impressive, you really need to know what the right answer is, because otherwise you can't judge how useful or correct the generated SQL is.
I have thought that LLMs might be great for someone who is already well-versed in the Assembly Language used to write software up to System 6, once it can be fed in appropriate texts, code and so on.

It can also same some time in disassembly analysis by identifying code blocks that are already well understood; however, I've found that there's a small enough body of work for pre-system 7 68k asm available that the current crop of LLMs have no clue what to do and keep on defaulting to OS X PPC/ObjC structures, and get confused about endianness and Pascal vs C structures. You'd need to do some very careful training to get a usable model there, I think. Post-2000 stuff? Yeah, it can save some time in basic scaffolding and in checking your work.

 

[olePigeon]

I write SQL for a living (which is sort of code) and while I often find LLM-generated SQL impressive, you really need to know what the right answer is, because otherwise you can't judge how useful or correct the generated SQL is.
I have thought that LLMs might be great for someone who is already well-versed in the Assembly Language used to write software up to System 6, once it can be fed in appropriate texts, code and so on.

I've been told by others that it does a great job commenting Assembly to make it easier to understand what's going on.

 

[olePigeon]

So I decided to start from scratch after a couple days of fiddling around. This time I set up parameters before I started and it's working a LOT better and producing compatible code way more often. It seems to be pulling its code almost exclusively from Julian Skidmore's "One Week Wonder" Blogpost which has a TON of posts with examples from programming in C that he'd done in the past. I really should write him an email and thank him. This program is 99% his code.

The parameters:

1. Use only C89/C90 standard for C programming.
2. Do not use Pascal.
3. Do not use Carbon.
4. Use only these following sources:
* The Macintosh C Programming Primer (Vol 1 & 2)
* Macintosh C Programming By Example
* Learn C on the Macintosh

 

[olePigeon]

OK. It's been nearly a month, and I've managed to accomplish what I suspect a competent programmer could have done in a weekend. Parts have been an immense struggle. The AI loves to hallucinate. I make a LOT of copies of code as I make short but stable progress. The problem I run into is that for some reason it will start to make random, arbitrary changes to other parts of the code completely unrelated to what I'm doing, completely breaking it. Even if I re-upload code from a stable build, it will insist that the random changes it made are exactly the same as the completely different code that I uploaded. I have to upload the same code file 3 or 4 tries to make the damn thing realize it's pulling completely random sh-t out of its ass instead of using the stable code I literally just uploaded as its source.

Absolutely infuriating.

In any event, I've finally managed to get to a major milestone in my app. For an extremely simple database program with less than a 1,000 lines of code, it has been a very slow grind. Holy cow. I don't think I'm learning much of anything. However, it's still really neat and fun to see an app I've always wanted to make come to fruition.

It continues to rely heavily on Mr. Skidmore's "One Week Wonder" blog. When it deviates, that's when it tends to mess everything up.

 

[David Cook]

Very interesting. I'm glad you're keeping us up to date.

I'm not surprised it has been difficult. I'm skeptical of LLMs for programming, for the same reasons other people have already mentioned in this thread. But, as a new way of searching and presenting findings, LLMs are wonderful. So, hearing about your progress is encouraging that there is still a way of making valuable use of LLMs in a programming environment.

I wonder if you incorporated additional Macintosh books and source code whether the results would improve.

 

[olePigeon]

So my app is now feature complete. This is the simple Reader, I do have an Editor, but since it's based off the Reader, I wanted to make sure the Reader app is mostly bug free before continuing off the same code base to make the Editor.

My app is an extremely simple database program with a very simple database file using custom delimiters. It is nearly identical to the Cracks & Numbers format, except that my SNDB format allows for a separate Cracks entry for each Version entry, and it also supports color text. I've also reimplemented a missing feature from Cracks & Numbers for support for patching applications, although not as cleaver. I figured since ResCompare already existed, just use that. Also, you can link serial number generators, hardware copy protection emulators, etc.

This was just something I've wanted to do since high school but never had the ability to do it, even if it doesn't really accomplish much more than what the existing C&N and S&K didn't already do.

At the very least, I had a lot of fun making the icons.

Feel free to run it through the ringer if you like. It was developed on an emulator, so there're probably some bugs I'd like to know about. I tested it on my IIfx and it seemed to run fine.

 

[olePigeon]

Very interesting. I'm glad you're keeping us up to date.

I'm not surprised it has been difficult. I'm skeptical of LLMs for programming, for the same reasons other people have already mentioned in this thread. But, as a new way of searching and presenting findings, LLMs are wonderful. So, hearing about your progress is encouraging that there is still a way of making valuable use of LLMs in a programming environment.

I wonder if you incorporated additional Macintosh books and source code whether the results would improve.

I see it as a potentially powerful tool for someone who is an experience programmer already. Maybe implemented as part of an IDE where it can make auto-complete suggestions using a code library of your own. That way the code will already be familiar (since it's based off code you wrote), and will have comments automagically generated in your own style so it's easy for someone else to understand what's going on. It could speed up coding pretty fast, but also provide suggestions for solutions you hadn't considered to a problem.

I think I would agree that it might improve the results, but also, it begins to hallucinate quickly. It doesn't matter how many times I tell it that it doesn't need #include <textutils.h> because it's included with the standard Mac library in THINK C, after a dozen or so prompts it will inevitably do it. Without fail. Only to respond, "You're right. You already told me this, but I included it anyway. I'll remove it."

 

[3lectr1cPPC]

I'm an amateur programmer, "learning as I go" and whatnot. I find LLMs helpful when I've got a bug I can't fix. I'll give it my code, ask it to tell me what's wrong, and I make sure I understand what the problem was rather than just blindly trusting anything. 9 times out of 10 it's one tiny syntax error that my eyes missed in the dozen times I scanned through the code myself.
That's about the limit of what I use it for though, troubleshooting my own code that I've writted. AI-generated code is problematic for a whole number of reasons which other posters here have laid out. But I think it can be a great learning tool to help inexperienced programmers, if they're using it correctly.

 

[olePigeon]

I wonder if you incorporated additional Macintosh books and source code whether the results would improve.

Do you have any recommendations for excellent books with good code examples for 68k programming?

 

[olePigeon]

tldr; I'm having fun realizing ideas I've literally had in my head for over 30 years but have been unable to do any of it.

--

My work re-enabled Chat History for staff, so I'm able to play with Gemini again. I've been maxing out my daily allotment of Pro & Think versions the past few weeks making silly programs for myself, but I thought I'd share them. I'll be posting their source code, too, if anyone wants to see what the LLM has stolen and how it's used.

I'm happy to say that KNPR (Kracks Numbers & Patches) has been updated to version 0.2a. It can now read SNDBs that are encoded in either BinHex or MacBinary. I did this to protect the 'styl' resource that's used to display color so that SNDB files can be freely distributed cross-platform. I gave much thought about just using a markup language to do color (such as ANSI escape codes) and actually went so far as to getting sidetracked into building an ANSI viewer program (you can see it working here), but using a markup language that almost nothing can read didn't seem very 80s/90s Mac-like. I also wanted the files to be easily viewable and editable on any vintage computer, so that ruled out RTFs.

I am currently finishing up a new utility to fill a gap in my archival process called DropHex (use with caution.) Currently I use a combination of DropBin and BinHex 4.0 to encode my files for archiving to my web archive, plus two more programs called FileTyper and Drop•Rename. Unfortunately I've run into two problems: BinHex 4.0 doesn't support Drag & Drop nor can you bulk encode. DropBin, on the hand, can do both those things, but it completely dies if a file name is longer than 31 characters or has the same name as a file that already exists. I deal a lot with files with really long names, especially disk images. As per tradition, I like to use ".image" for Disk Copy 4 images, which can quickly lead to file names that exceed 31 characters when you start adding ".hqx" to the end.

DropHex supports Drag & Drop, bulk encoding files, truncating long file names, prompts for duplicate file names, options for Save locations (same as file, prompt, or specific folder), option to set Type & Creator to Stuffit Expander, and add Line Feeds for PC compatibility.

It's only been tested in an emulator at the moment, so use with caution. I'll need to do more testing. I also need to see if I can optimize performance on encoding/decoding, including making use of FPUs if they're available.

Here're the nifty icons I made for it.



MultiBin is a feature I've removed, but might put back in? I don't know. It seemed antithesis to the original BinHex and single-file encoding. It's basically just multiple binhexed files stitched together with some info on directory structure if you have folders. That's it. I guess it's an uncompressed plain text archive at that point. Although you could literally open a MultiBin file in SimpleText and copy/paste each file back into their constituent files if you wanted to. Or use DropHex to decode them (and their folders.)

Another program I'm working on (but I'm not posting it yet, see below) is DropSTAC. DropSTAC is a compression utility that uses the original STAC LZS compression algorithm used in their prolific STAC 9703 (and others) microprocessor used mostly in tape drives, but also with real-time compression cards for PCs and Macs. But "WHY?!" I can hear you yelling, "we already have Stuffit or working open-source LZS derivatives like LZW!" Because I'm old and bitter, and I'm beyond disappointed that I can't use my DoubleUp real-time compression card to make archives that can be uncompressed without the card, that's why! There's the original ISA based Stacker hardware accelerator/real-time compression card for PCs (I'm sure there were a bunch more 3rd party competitors), and at least three real-time compression cards for the Macintosh that I'm aware of, all of them incompatible with each other. I figured it'd be cool to use an exceptionally well-documented compression algorithm (something that could be easily reverse engineered in the future) and, more importantly, I have an end-goal of getting it to work with my DoubleUp card. DiskDoubler with a DoubleUp card is very cool, but I suspect they were hamstrung by some sort of licensing deal with STAC. Making an self-extracting archive that was technically an application itself probably required additional licensing on a per-app basis that's not practical for something that's intended to be shared freely, so they locked it to a hardware card. I guess technically that makes the DoubleUp card a hardware copy protection device.

I am currently working on it, and I do plan on finishing it. What I'm uncertain about is the legality of releasing it. Even though (I think) it's an ANSI standard, I think the algorithm is still copyrighted. The archive format is just MultiBin (see DropHex above) compressed with the STAC algorithm. Very simple, well-documented already, and easy to reverse engineer in the future.

Anyway, here're the icons I had fun making.

 

[bigmessowires]

The AI loves to hallucinate. I make a LOT of copies of code as I make short but stable progress. The problem I run into is that for some reason it will start to make random, arbitrary changes to other parts of the code completely unrelated to what I'm doing, completely breaking it.

I'm glad you're having fun. Cool tools! What you describe about random arbitrary changes has also been my experience when using a generic AI like Gemini or ChatGPT for programming. It's maddening. But when using a coding-specific AI tool it's mostly a different story. I went from tearing my hair out with ChatGPT (OK I'm already bald, but metaphorical hair) to a relatively civilized experience using Visual Studio Code with the Gemini Code Assist plugin - both free. I hear that Claude Code is even better. The key is that the AI is part of your programming environment, it has visibility into all of the files and build settings, as opposed to just copy-pasting code blocks into a typical AI chat session.

Is it flawless? No. But it's shockingly capable. Capable enough that I'm glad I already left my career as a professional software developer, because this tool can do big pieces of what I used to get paid a lot of money for. My jaw dropped the first time that I described a new feature with a few sentences worth of underspecified description, and it just built it, making reasonable assumptions about the parts I hadn't mentioned, and it all worked perfectly the first time.

 

[Durosity]

tldr; I'm having fun realizing ideas I've literally had in my head for over 30 years but have been unable to do any of it.

This. As a general concept I’m not particularly enamoured with AI. I think that it’s creating way more problems than it solves (energy usage, unauthorised image manipulation of young women, and just general laziness when it comes to topics that really deserve proper research as just a few examples).. but where I think it does shine is enabling those of us who have never been able to master programming to “create” and finally be able to contribute towards tools and utilities and games that can benefit many people.

For example, I’m a massive fan of the home automation package Indigo (Home Assistant sucks!!), but beyond a few simple python scripts I was never able to give much back to the community who use the system. Now with AI I’ve been able to create 15 plugins which give it functionality I’ve wanted to see for literally decades.

I appreciate there’s a LOT of moral considerations, but it can be used for good. ¯\_(ツ)_/¯