MacSE-RAM, a crazy Mac SE PDS RAM concept!

[cheesestraws]

I'm afraid there are lots of places in the OS, especially in early boot, where it assumes the RAM is all contiguous. Once the memory manager is started, yes, you might be able to get away with permanent allocations, but getting it to the point where the memory manager is working will be more challenging. You might be better off moving the ROM and SCSI mappings to the end of your revised memory map - software already has to cope with those being at basically arbitrary addresses because different models of Mac have different mappings, and although it sounds like more drastic surgery, it probably actually isn't...

edit: @SuperSVGA beat me, but yeah, what they said.

 

[joevt]

Memory Management is described at
https://developer.apple.com/library/archive/documentation/mac/pdf/Memory/Intro_to_Mem_Mgmt.pdf
I think the unusable parts of RAM would need to be disguised as entire Application partitions? But that won't work if something needs to write to parts of the Application partitions...

The problem is, that an Application partition that includes unusable parts of RAM needs to span the unusable part of RAM in such a way that the Heap part of the partition spans the entire unusable part of RAM so that it can include the unusable parts as a memory block and so that the stack cannot collide with the unusable part.

 

[Big Ben]

Some of us have a Levco accelerator which does use on board RAM and accelerator RAM.

On board RAM is used as RAM disk only if I’m not mistaken, because it is slower than the one used by the accelerator.

This accelerator use a ROM and GALs to patch the macintosh ROM, we dumped the first but not the later. From what I read, Levco would have added extended memory support ans SuperDrive compatibility by providing a new set of GALs chips. I don’t think it was ever released.

Maybe there is a lot to learn from this board that could be useful here.

I can try to dump the GALs if given the procedure.

 

[cheesestraws]

Yeah, that seems to have been a relatively well-distributed, if slightly bonkers, technique. I remember @demik talking about it too...

 

[Snial]

I have a prototype open-source 68HC000-based accelerator made in partnership with Garrett’s Workshop that can do what’s described here. The accelerator has 4 MB of onboard fast RAM for the main 4 MB of RAM at $000000-$3FFFFF and does not use motherboard RAM except for the video/sound buffers. Since the motherboard RAM is mostly unused, the accelerator can be configured to remap the addresses in the $600000-$8FFFFF range to motherboard RAM. That way you get almost 3 MB of additional RAM but due to limitations of the existing hardware, there’s a 64 kB hole in the middle where the screen and sound buffers are. So you get 1.9375 MB contiguous, then 64k screen/sound, and then another contiguous 1 MB. Won't be that fast but the accelerator (unlike many legacy designs) is able to do back-to-back reads from the motherboard and can also do a posted write to motherboard RAM. So even when running an app out of motherboard RAM, it will be significantly faster than on an unaccelerated SE. When I brought up the accelerator initially, I disabled onboard RAM and used motherboard RAM exclusively to ensure the bus bridge to the motherboard had good performance. In this case Speedometer 3 was giving something like 1.25 CPU score compared to a Mac SE baseline score of 1.0. Running from fast RAM exclusively on the accelerator gets you like 4.15x CPU score. Since the main 4 MB of RAM on the accelerator contains the OS, lowmem globals, and framebuffer, running a typical app from motherboard RAM will be even faster than 1.25x but almost certainly less than 2x except in contrived cases.

I have a thought about patching the OS to use this extra RAM. It’s totally uninformed except from a theoretical perspective. The Mac’s memory manager probably maintains a free chain, right? This is a linked list representing allocated and free regions of memory. I guess at the system level it would be representing the heap spaces for different apps and the system. When more RAM needs to be allocated, the memory manager looks through the free chain and finds a free region of memory large enough to make the allocation and then updates the free chain with a new linked list item saying that the memory in that region has been allocated. At least, this is how it usually works. Can’t we just find the free chain and make some more entries to represent the permanent allocation that is ROM and SCSI at $400000-$5FFFFF and the free RAM at $600000-$8EFFFF? I’d try that first but there are a number of reasons that it might not work.

I just built some new prototypes but have yet to try them out. I just updated the code to map the memory at $600000-$8FFFFF as I described. If anyone formulates a good plan to patch the memory manager I’ll send you a prototype WarpSE to develop it on. If it can be made to work well, I will upgrade future versions of the WarpSE with more fast RAM so that high memory is just as fast as regular RAM and also fix the memory mapping so there isn't the hole for screen/sound RAM in the middle.

Wow, so basically you've worked on something like this! I guess that the fastest way to check this is to modify miniVMac to emulate WarpSE (and maybe the model should be called WarpSE), then (b) we can write any INITs or extensions (pretty much the same thing, right?).

I'm guessing, but I think @joevt , probably understands more about non-virtual memory management on compact Macs, that we might need 2 OS INITs. The early one tells the Mac it has RAM from $000000 to $8FFFFF. It can't set up application partitions, because the memory manager (which essentially is MultiFinder on the SE even for System 7 isn't it?) won't have created any yet. In addition, it hijacks memory clear routines to just RTS if there's any attempt to write to the memory hole from $400000 to $5FFFFF. It doesn't matter if it slows down Mac initialisation, because once we've set up the partitions, the routines will be unhijacked.

Then we let the rest of System 5, 6 or 7 boot.

The second part will be an application itself in System 7 in the Start-up Items folder (or Set-Startup on System 6). @mihai asked about this last year https://68kmla.org/bb/index.php?threads/startup-items-equivalent-under-system-6-multifinder.43175/ .

That application allocates the hole in the heap, either directly or by generating dummy applications. E.g. You allocate a dummy application down to $600000; then allocate the MacSEHole.app from $400000-Size(MacSEHole.app) to $5FFFFF. MacSEHole's size needs to be big enough for a stack and whatever A5 space it needs. Finally, the first dummy application quits followed by the start-up application.

Memory mapper on System 7.1, Quadra 650:

 

[Snial]

Wow, so basically you've worked on something like this! I guess that the fastest way to check this is to modify miniVMac to emulate WarpSE (and maybe the model should be called WarpSE), then (b) we can write any INITs or extensions (pretty much the same thing, right?).

I'm guessing, but I think @joevt , probably understands more about non-virtual memory management on compact Macs, that we might need 2 OS INITs. The early one tells the Mac it has RAM from $000000 to $8FFFFF. It can't set up application partitions, because the memory manager (which essentially is MultiFinder on the SE even for System 7 isn't it?) won't have created any yet. In addition, it hijacks memory clear routines to just RTS if there's any attempt to write to the memory hole from $400000 to $5FFFFF. It doesn't matter if it slows down Mac initialisation, because once we've set up the partitions, the routines will be unhijacked.

Then we let the rest of System 5, 6 or 7 boot.

The second part will be an application itself in System 7 in the Start-up Items folder (or Set-Startup on System 6). @mihai asked about this last year https://68kmla.org/bb/index.php?threads/startup-items-equivalent-under-system-6-multifinder.43175/ .

That application allocates the hole in the heap, either directly or by generating dummy applications. E.g. You allocate a dummy application down to $600000; then allocate the MacSEHole.app from $400000-Size(MacSEHole.app) to $5FFFFF. MacSEHole's size needs to be big enough for a stack and whatever A5 space it needs. Finally, the first dummy application quits followed by the start-up application.

Memory mapper on System 7.1, Quadra 650:
View attachment 77971

Actually, the Startup app can be simplified so that it allocates just enough space above $600000 for the MacSEHole's stack and A5 world; and just enough space below $400000 for the MacSEHole's heap and Code segment 1, which I think is not much more than a GetNextEvent or WaitNextEvent loop so you can switch back to a real app.

-cheers from Julz

 

[ZaneKaminski]

Do we have to have an INIT or ROM patch at all? It seems worse to make RAM contiguous at boot because then "highmem," A5 world, and the stack end up in the slow RAM. Instead we should let that stuff go from 0x3FFFFF downwards at boot. Then after the system is fully booted we should patch the memory manager to allocate application heap zones in 0x600000-0x8FFFFF. At that point we can also patch BufPtr and MemTop in case anything refuses to work when allocated above those locations.

Edit: I guess MemTop is a per-application variable that points to the top of an app's heap zone, so maybe all that needs patched is BufPtr, if even that. Does anything use BufPtr-relative addressing? In that case we can't patch BufPtr. After boot, what uses BufPtr anyway?

 

[ZaneKaminski]

Okay I guess A5 world and obviously the stack are app specific, but still, whatever goes in this highmem area would probably benefit from being in the fast RAM. I just read the memory management chapter in Inside Macintosh but it doesn't say much about how the OS creates the individual application partitions.

 

[SuperSVGA]

You could always put all of the memory space in the "fast RAM" and only access the logic board RAM when needed.

 

[joevt]

I don't think we want an app to cover the holes. That would only be good for being able to run more small apps. What if you have an app that you want to assign > 4 MB to?

So you need something to make sure the holes don't get used (for temporary memory allocations or whatever) when no App partition is covering them. And then you need something to make sure the application partition is large enough to include the holes and a stack (increase some of the numbers that originate from the SIZE resource if necessary).

It appears that Application partitions begin from BufPtr. This allows the System Heap to grow upward if necessary as long as there's no Application partition in the way.

Will the patches be made before or after the first App is loaded? Will the first App be loaded below 4MB or below 9MB? Can the first app be relaunched to move it?

 

[Snial]

Do we have to have an INIT or ROM patch at all? It seems worse to make RAM contiguous at boot because then "highmem," A5 world, and the stack end up in the slow RAM. Instead we should let that stuff go from 0x3FFFFF downwards at boot. Then after the system is fully booted we should patch the memory manager to allocate application heap zones in 0x600000-0x8FFFFF. At that point we can also patch BufPtr and MemTop in case anything refuses to work when allocated above those locations.

Edit: I guess MemTop is a per-application variable that points to the top of an app's heap zone, so maybe all that needs patched is BufPtr, if even that. Does anything use BufPtr-relative addressing? In that case we can't patch BufPtr. After boot, what uses BufPtr anyway?

There's another queue to manage all the heap zones in the Mac's memory. It's probably another linked list, but there's also some heap-checking. I don't imagine it'd be horrific, but one would need to take care. Global variables are listed here:

Linux/m68k for Macintosh - Mac Alamanac II

 

[Snial]

I don't think we want an app to cover the holes. That would only be good for being able to run more small apps.

Yes, that's the idea. You can run more apps, but not big apps. That's still useful. One app, for example could be a RAM disk, then you get the RAM disk option too.

What if you have an app that you want to assign > 4 MB to?

You can't, but on System 7.1 you wouldn't even be able to run a 3MB app as the OS + screen will take more than that, but that should be possible with this scheme.

So you need something to make sure the holes don't get used (for temporary memory allocations or whatever) when no App partition is covering them. And then you need something to make sure the application partition is large enough to include the holes and a stack (increase some of the numbers that originate from the SIZE resource if necessary).

The application partition approach is the simplest one I can think of, even though it prevents large apps from running (just more, smaller ones).

It appears that Application partitions begin from BufPtr. This allows the System Heap to grow upward if necessary as long as there's no Application partition in the way.

Will the patches be made before or after the first App is loaded? Will the first App be loaded below 4MB or below 9MB? Can the first app be relaunched to move it?

So, if it's a startup app, then I guess it'd work downwards from 4MB. This is why I suspect an INIT is needed to first make MemTop $8FFFFF; then it quits and releases it's memory; then at the end, the startup app runs; which creates the application partition to create an app above the hole; another application partition to cover the hole; then quits the app above the hole and itself.

I'm only trying to think of the dumbest, simplest method for now: a proof-of-concept.

 

[joevt]

The application partition approach is the simplest one I can think of, even though it prevents large apps from running (just more, smaller ones).

If you learn how to create an application partition at a specific location and size, then you have everything you need to create an application partition of a larger size with memory allocations that cover the hole(s).

So, if it's a startup app, then I guess it'd work downwards from 4MB. This is why I suspect an INIT is needed to first make MemTop $8FFFFF; then it quits and releases it's memory; then at the end, the startup app runs; which creates the application partition to create an app above the hole; another application partition to cover the hole; then quits the app above the hole and itself.

I'm only trying to think of the dumbest, simplest method for now: a proof-of-concept.

An INIT works in the system heap. I suppose it would make some patches that affect how application partitions are created. An INIT doesn't quit. It might unload parts of itself it no longer needs (the part that draws the startup icon or makes the patches).

 

[ZaneKaminski]

If you learn how to create an application partition at a specific location and size, then you have everything you need to create an application partition of a larger size with memory allocations that cover the hole(s).

I don't think it's a good idea to try and put a hole right in the middle of an application partition. I understand you can do this by putting a big nonrelocatable block in the middle right before the app starts running, but semantically it's not in line with the application lifecycle model. The application partition should start out totally unfragmented. If there's stuff in the middle of the application partition, the app may not expect this and could crash or lead to buggy behavior. It just amounts to pushing the fragmentation problem from the OS down to the app, which totally doesn't expect it. Plus nobody needs to allocate 4 MB to a single app on a 68000-based Mac. Even 1 or 1.5 MB is a large app for the SE. I think my Perofrma 6214CD I had as a kid was equipped with only 8 MB and it had a PowerPC 603e.

In the future we can fix the "hole" at $7FXXXX by rearrangeing the memory mapping in the hardware and putting the video and sound buffer area at $8FXXXX instead. The current hardware can't do this for idiosyncratic reasons. In the meantime, it would suffice just to figure out how to get the system allocating application zones in the $600000-$7EFFFF area. We can ignore the $8XXXXX space until later when we improve the memory mapping.

Too bad applications are allocated space starting from the top of RAM downward. This would put the Finder and whatever else you start up after boot in the slow motherboard RAM. I forgot that detail about the system heap needing to expand and figured the allocations would start from the bottom of RAM after the system heap.

Anyway, these obstacles are best addressed with hardware changes to improve the RAM mapping and speed. In the meantime we could just focus on making the system allocate application zones in the $600000-$7EFFFF area.

 

[ZaneKaminski]

Seems like the code for starting an application is mostly in Processes.c (from the leaked "Super Mario" sources): https://github.com/elliotnunn/super...erMarioProj.1994-02-09/ProcessMgr/Processes.c

NewProcess(...) calls GetProcessMemory(...) and this allocates the application zone using ProcessMgrHiNewHandle(...), which returns a handle just like NewHandle(...) in an app. It even uses Lock to make the application zone nonrelocatable. Interesting! It's memory managers all the way down hahahah.

MemoryMgrPatches.c has the implementation of ProcessMgrHiNewHandle(...) and some more good stuff: https://github.com/elliotnunn/super...Proj.1994-02-09/ProcessMgr/MemoryMgrPatches.c

Check this out from MemoryMgrPatches.c:

/* InitHeaps.  Initialize the system and Process Mgr heaps.  The fundamental change we
 * implement is a flexible boundary between the system zone and the Process Mgr zone.
 * To do this, we ensure that non-relocatable or locked blocks occur low in the
 * system heap, or high in the Process Mgr heap.  We start the ball rolling here by
 * a) setting a handy flag to force the SYSZONE allocations low, b) moving existing
 * relocatable blocks out of the Process Mgr heap, c) relocating the Process Mgr master
 * pointer block high, and d) taking over the grow zone procedures of the two heaps.
 * The original Process Mgr MP block is turned into a minimum size pointer block at the
 * bottom of the heap.  The system grow zone procedure will use a SetPtrSize on this
 * block to get enough space to extend its zone into the bottom of the Process Mgr zone.
 *
 * NOTE: Error checking should be done in here!
 * A5 world needed:  PROCESSMGRGLOBALS.
 */

Well that's a useful explanation. Back to ProcessMgrHiNewHandle(...), the first thing it does is call SafeSetZone(theZone, ProcessMgrZone). I guess ProcessMgrZone is a global variable pointer to a Zone struct? My takeaway from all this is that the process manager heap stores application heaps and the usual style of Mac OS memory management applies to the process manager heap same as within each application heap.

 

[Snial]

Seems like the code for starting an application is mostly in Processes.c (from the leaked "Super Mario" sources): https://github.com/elliotnunn/super...erMarioProj.1994-02-09/ProcessMgr/Processes.c

NewProcess(...) calls GetProcessMemory(...) and this allocates the application zone using ProcessMgrHiNewHandle(...), which returns a handle just like NewHandle(...) in an app. It even uses Lock to make the application zone nonrelocatable. Interesting! It's memory managers all the way down hahahah.

MemoryMgrPatches.c has the implementation of ProcessMgrHiNewHandle(...) and some more good stuff: https://github.com/elliotnunn/super...Proj.1994-02-09/ProcessMgr/MemoryMgrPatches.c

Check this out from MemoryMgrPatches.c:

/* InitHeaps.  Initialize the system and Process Mgr heaps.  The fundamental change we
 * implement is a flexible boundary between the system zone and the Process Mgr zone.
 * To do this, we ensure that non-relocatable or locked blocks occur low in the
 * system heap, or high in the Process Mgr heap.  We start the ball rolling here by
 * a) setting a handy flag to force the SYSZONE allocations low, b) moving existing
 * relocatable blocks out of the Process Mgr heap, c) relocating the Process Mgr master
 * pointer block high, and d) taking over the grow zone procedures of the two heaps.
 * The original Process Mgr MP block is turned into a minimum size pointer block at the
 * bottom of the heap.  The system grow zone procedure will use a SetPtrSize on this
 * block to get enough space to extend its zone into the bottom of the Process Mgr zone.
 *
 * NOTE: Error checking should be done in here!
 * A5 world needed:  PROCESSMGRGLOBALS.
 */

Well that's a useful explanation. Back to ProcessMgrHiNewHandle(...), the first thing it does is call SafeSetZone(theZone, ProcessMgrZone). I guess ProcessMgrZone is a global variable pointer to a Zone struct? My takeaway from all this is that the process manager heap stores application heaps and the usual style of Mac OS memory management applies to the process manager heap same as within each application heap.

Makes sense. I was reading the process Manager documentation today.

Processes (IM: Pr)

dev.os9.ca

 

[demik]

Yeah, that seems to have been a relatively well-distributed, if slightly bonkers, technique. I remember @demik talking about it too...

Indeed. but as @Melkhior that requires a working MMU so a 68030. Interested to see if you manage to find a way for this on a bone stock 68000…

 

[ZaneKaminski]

Accelerator is working with the new memory mapping:

Sorta obtuse to read but the gist is that the A23 and A22 address lines going from the accelerator to the Mac's motherboard are forced to 0 when the address is in the $6XXXXX, $700000-$7EFFFF, and $8XXXXX range. This maps these ranges to RAM on the motherboard. The current hardware can't do anything with A21 and A20 so we can't map $YXXXXX to $ZXXXXX for any Y and Z in the range $0-$F but I could add some new wires in to accomplish that in the future if desired. That would let us avoid having the video/sound hole at $7FXXXX.

So I would say that having the MMU is not really that important for getting more RAM on an accelerator. MMU gets you fully programmable address mapping that the OS can change whenever, but we just need a fixed mapping. Anything like that can be implemented in the accelerator controller, including moving ROM/SCSI elsewhere. The issue is the software.

On that note, for 68030 accelerators in the SE, there's Connectix Compact Virtual 3.0, which uses the MMU to map all the I/O near the top of the address space. That way you can get almost 16 MB of contiguous RAM. I have never tried Compact Virtual but I hear that when its INIT runs during boot, it rewrites the various I/O pointers in lowmem to point to the new locations near the top of RAM, then changes the memory mapping, and then initiates some kind of partial reboot. Interesting. I snooped around in the Compact Virtual code using ResEdit and its disassembler but couldn't find where the part that did this was. Maybe someone could produce a disassembly of this key part of Compact Virtual so it could be studied.

 

[Snial]

Makes sense. I was reading the process Manager documentation today.

Processes (IM: Pr)

dev.os9.ca

@ZaneKaminski , @joevt. I've done the first part of my objectives for this. It's very crude. Application is attached. It's written in THINK C and I developed it on miniVMac emulating a 4MB Mac Plus running System 7.0. I thought I should specify this in case it doesn't work on other setups yet.

AppGap, like I said is crude. It's a tiny 3kB program which I duplicated. By default THINK C allocates 384kB to the application partition. I made a duplicate (AppGap2) which has an 80kB allocation. When AppGap is run it displays its ProcessAddress and Length. Then you can use File:Open to open an application (I always select AppGap2).

AppGap allocates exactly the right temporary handle (which it HLocks to stop the memory manager from moving it) so that when the selected application is launched, there's enough space for its code segment below the intended starting address of 1.5MB + a gap of 64kB + 8kB for the stack above the gap. I might need to adjust these figures, but the principle holds, I believe.

Quitting AppGap, leaves AppGap2 running with the same larges unused block size (though stupidly I didn't capture that on the screen):

To prove it allocates at an absolute address, I ran WriteNow with an application partition size of 1024KB; then did the same process.

AppGap2 was still allocated at 1564304. Quitting AppGap left AppGap2 at the same location (I kept forcing a window update to prove it was still at the same address):


The program is attached, along with the source code. The next steps will be to make a large memory SE believe it has memory up to the address we talked about earlier; and use a modified version of AppGap with the gap address set to 4MB instead of 1.5MB and the gap 1MB instead of 64kB.

-cheers from Julz

 

[zone66x]

Maybe this would be somewhat relevant to the discussion:

https://github.com/simon-frankau/big-classic-mac