October 2021 Project: reverse-engineering Studio Session song format

[Mu0n]

My project is not new, it's a continuation of something I pushed forward in 2019 back in another thread:

Studio Session File Format reverse-engineer

If you're not familiar with Bogas Productions' 1986 Studio Session app, then check out this other thread I made: Reverse-Engineer Goal: Load a Studio Session song file (along with its instruments) and play it back, close to the original in a custom game program (eg in an intro splash...

68kmla.org

The goal of retrochallenge is to advance and/or complete projects, right and talk about some milestones, right? Here goes.

Goal: Load a Studio Session song file (along with its instruments) and play it back, close to the original in a custom game program (eg in an intro splash screen)

Context: Studio Session from Bogas Productions released in 1986. It uses 11 kHz instrument sound samples of arbitrary sizes and mixes up to 6 tracks together to create music that went beyond the default Apple Sound Driver, whose most similar mode of usage featured 4 channels of 256 byte-long sound waves.

Official website: https://mu0n.github.io/ForayInto68k/ress/

Code: - available when phase 1 is mostly done, but will be here: https://github.com/Mu0n/ForayInto68k -

Use case if the project comes to completion: This song format has a very mature song editor with the usual Mac GUI frills using standard musical notation. There are nifty tools that allows to re-use chunks of music (phrase library) and repeat some sections. If I can programmatically play back a song, this could be incorporated into a game programming project very easily. Some advanced music could be played back in a title screen or during a non-cpu intensive part of a game (cinematics for example). This was done commercially in some games like Tetris, Falcon F-16 and Vette!, all from Spectrum Holobytes. The code they used would be what the final form of this project strives to reach.

Target Hardware Environment: same machines that used to run Studio Session, which is probably a stock Mac Plus with a 68000 and 1Mb of RAM

Target Software Environment: A System version that supports the Sound Driver at first. If all goes well, target the next few versions of System which made use of the Sound Manager. I suspect this is a high step between these 2.

Programming Tools: Symantec THINK C 6.0, THINK Reference, ResEdit, SoundCap (or others), Studio Session

Phase 1 (in progress):
---DONE:
-Load a song file
-Detect basic info: instrument list, master key, master tempo, most proprietary commands
-Use the Sound Driver square-wave synth to play back a channel
---TO DO:
-Create an interface that displays song info
-Select a channel to play back
-Very aggressively cut up the source code into many small files for future management
-Get some timing measurement routines to test out some optimizations

Phase 2:
---TO DO:
-Use the 4-synth driver to select and play 4 channels at once
-Figure out how the notes are synchronized together, ie, what's the smallest time slice where note changes happen, instruments come in/out, etc.
-Figure out buffering used during playback - can the whole song be loaded at once, or is it done on the fly
-Figure out if sync needs to happen with video blanking (VBL) interrupts

Phase 3:
--TO DO:
-Figure out how much memory is needed
-Figure out a mixing method to avoid saturation of the freeform synth driver - divide by 2, sqrt(2) or 6?
-Re-figure out buffering used during playback - can the whole song be loaded at once, or is it done on the fly
-Optimize data transfers to squeeze out some free CPU cycles as much as possible
-Explore using ASM blocks for some critical parts of the program as needed
-Find a good pipeline for making new instrument sound files

 

[Crutch]

Still a great project. Though I’d love to just reverse engineer how it’s done in Tetris. There must just be a VBL task or driver running that plays Studio Session sound files. It would be fun to figure out where that code is and just pluck it out.

 

[Mu0n]

Things done in the first week:

-Plunge back into 2 year old code
-Chop down the main C file very aggressively into many sub files (DealWithMouse.c, DealWithKeyboard.c, Menu.c, Loadsong.c, etc)
-Retool my brain away from Java classes and into using a bunch of #pragma once lines everywhere, as little few globals as needed and efficient and minimal cross talk between my source files
-About screen as an alert (as seen above in the image)
-Working menu events
-Refresh on the usage of a resource file tied to the THINK C project - it's currently driving the window, menus and alert

 

[Mu0n]

Huge snags hit as my semester goes from full time teaching to around 125% load because of a colleague taking a leave of absence (her workload is split between 6 full time teachers) before we scramble to hire someone new (hint: October is the lowest of lulls to hire someone new).

But my code is CHOPPED UP way more than before. Communication between functions is kept TIGHT. The number of globals needed is heavily diminished. One trick that could help me greatly is to pass signals through events, but I'm not sure why it's not working right now.

@Crutch you usually have my back in these pointed technical questions. Inside a source file away from main, after I'm done siphoning the song file data, after passing all tests, I'd like to signal the main loop to populate some info on screen. I'm using PostEvent(app3Evt,0) (with no meaningful info, I just want a trigger). In the main loop of my main c file, I would want to detect that event and interpret its data and refresh the UI. However, it's not reacting at all. In THINK Reference, they tell you to avoid app1Evt and app2Evt, hence why I'm using app3Evt, but I tried the other ones and no cigar still. I know this is not supposed to be deprecated by System 7, but it doesn't work anyway in System 6.

What would you use instead for this? I know I can boolean global flags but I'd rather keep this cleaner.

EDIT - NVM! I put the reaction to the event in a commented out function. More cleaning left to do (it confirms I'm justified into heavily reorganizing stuff). Still, would you adopt this method as viable?

 

[Crutch]

Glad you fixed the issue!

It is tempting to use PostEvent but it is generally not a good idea. The problem is that under MultiFinder or System 7, the frontmost application always gets the next event. So if you post your app3Evt while your app is running in the background (like, the user accidentally clicked outside of one of your windows, bringing the Finder back to the front), the new front application will get an app3Evt, not your application — and that event will of course be removed from the event queue, so you will in fact never see it.

So unless you are only running in a non-multitasking environment (System 6 or below, without MultiFinder), using PostEvent can have unpredictable results. I would probably use a global flag for your use case, inelegant as it may seem!

See Tech Note 180, “Multifinder Miscellanea” for a few details.

 

[cheesestraws]

A global probably is the simplest here, yes; one has an inbuilt reaction away from them but sometimes they are the simplest option. A slightly more elegant version would be to have a struct somewhere with any flags needed for communication, then pass a pointer to that to both routines. But you then have to pass that around, and if you're only orchestrating one instance of two communicating routines, then it's probably not worth the overhead, and a global will actually be slightly neater.

 

[Mu0n]

Plenty of rust formed over Handles. I remembered their purpose very well with regard to the memory management strategy of the Mac. While I may not necessarily need to have a relocatable patch of memory in the heap for my song file content, I might need it later for the collection of loaded instrument sound files and their mixed buffer later down the road, especially if the venerable 1 Mb Mac Plus is my main target.

I'm currently not understanding what I'm seeing. After loading a song file, I get its content with:

err = GetEOF(fRefNum, &fileSize);    //get the size of a file opened with SFGet, which gave me a fRefNum\
gHandleFileContent = NewHandle(fileSize); //allocates RAM to the content handle
if(gHandleFileContent == 0) errorCantAllocate();
HLock(gHandleFileContent);
FSRead(fRefNum,&fileSize,*gHandleFileContent);      //put the content into extern global file content buffer
HUnlock(gHandleFileContent);

At this point, if I check the value of fileSize, I get 2487 (0x09B7) for the song 'Cubano', which reflects what I'm seeing in an hex dump of the file (although the last byte is 0x09B6? Is there a reason it's off by 1 byte?)


And now, the weirdest thing. Believe me I've tried every variant I remembered, counter checked with the page on Handles in THINK Reference. I'm trying to get simple data from the header of that song file at specific, known positions. The first 2 bytes contain the tempo information. The useful values recognized by the Studio Session format go from 10 to 450, so you need 2 bytes for this. Cubano has byte 0 at 0x00 and byte 1 as 0x96, for a decimal tempo of 150. When I was using FSRead to pick out this and other basic song data piece by piece using FSRead multiple times, advancing at a pace of 1 or 2 bytes at a time, there was no issue at all. I just want to do it at a later time, arbitrarily often, by reading it off a handle pointing to the read file's content long after the file is closed.

//Try #1
return **gHandleFileContent; //returns 2487 again (???)
//Try #2
return **gHandleFileContent[0]; //compiler complains about 'Pointer required'
                         // despite being an example in THINK Reference of an access to the first byte of data under the handle umbrella
//Try #3
char *ptr;
short result;
ptr = *gHandleFileContent;
HLock(gHandleFileContent);
result = *ptr << 8;
ptr++;
result |= *ptr;
HUnlock(gHandleFileContent);
return result; // still outputs 2487

If I dump the first 20 bytes as (10) characters straight up from gHandleFileContent, I get the same thing I see in the file's hex dump (BBEdit):




TLDR; I'd like to understand what I'm not doing right with handles, not because I *need* it for this particular task, but because I'll probably need it in the long run.

 

[Crutch]

First the easy bit:

GetEOF() returns the logical end of file, which is defined to be one more than the zero-based number of the last byte in the file. This is precisely so you can do things like call GetEOF() then allocate a handle of the corresponding size (if you have a 10-byte file, the bytes are numbered 0 through 9 but you need to allocate ten chars to hold them all).

 

[cheesestraws]

(although the last byte is 0x09B6? Is there a reason it's off by 1 byte?)

Because the first byte is labelled byte 0, so the last byte will be 0x09B6 if there are 0x09B7 bytes (a smaller example: there are four numbers in the list [0, 1, 2, 3] but the last number is 3 because the first is 0)

 

[Crutch]

Your code looks OK. I have a weird theory, but first, are you checking to see that fileSize is unchanged by the call to FSRead()? (It should return the actual number of bytes read, you should always check.) You should also check the OSErr code returned by FSRead().

Now my weird theory. You omitted your variable declarations. You aren’t by chance declaring fileSize as a (short) int instead of a long? If you are, then pass its address to FSRead() (which thinks it’s getting a pointer to a long, which it then modifies), the low-order word would get written down by FSRead() in the next word in memory after wherever fileSize lives, and it’s possible this is the first word of the relocatable block allocated by NewHandle(), meaning your buffer will start with the file length. It’s a long-shot explanation but the only thing I can think of instantly. And it sort of clicks with the fact that you report fileSize as “0x09B7” when it should be a long reported by the debugger as 0x000009B7.

 

[cheesestraws]

return **gHandleFileContent[0];

I think you want here (*gHandleFileContent)[0].

(this won't have any effect on your weird data corruption, just this is probably why the compiler is complaining)

 

[Crutch]

(Yeah that’s an inadvertent triple dereference as written, for the record I was disregarding that when I said code looks OK. )

 

[Mu0n]

Your code looks OK. I have a weird theory, but first, are you checking to see that fileSize is unchanged by the call to FSRead()? (It should return the actual number of bytes read, you should always check.) You should also check the OSErr code returned by FSRead().

Now my weird theory. You omitted your variable declarations. You aren’t by chance declaring fileSize as a (short) int instead of a long? If you are, then pass its address to FSRead() (which thinks it’s getting a pointer to a long, which it then modifies), the low-order word would get written down by FSRead() in the next word in memory after wherever fileSize lives, and it’s possible this is the first word of the relocatable block allocated by NewHandle(), meaning your buffer will start with the file length. It’s a long-shot explanation but the only thing I can think of instantly. And it sort of clicks with the fact that you report fileSize as “0x09B7” when it should be a long reported by the debugger as 0x000009B7.

FSGetFile has to report a good file record, FSOpen and FSRead have to return a noErr to get out of this loading block of code.

fileSize is a long. I did check it with my debugger and I learned the hard way that an int wasn't enough, but I fixed it before my thead.
As for 0x09B7, that's just my own writing, I didn't mean that it wasn't a long. I got this by sprintf'ing the value with a %X type into a C string before passing it to a Pascal string before DrawString (a cheap, but effective debug method) - what it wrote was simply 9B7. Good point about the logical vs physical size. Do you recommend I assign the physical size to my handle, to avoid the occasional extra byte of unneeded memory? Handles are byte aligned, right?

I just rechecked the fileSize after FSRead is done, no change (still 2487).

 

[Mu0n]

I found the error!

The condition that had to be met before I could extract the first 2 bytes to get the tempo was this:

if(GetHandleSize(gHandleFileContent) ==noErr)

The mixup is that I thought noErr was returned by GetHandleSize instead of MemError (which wasn't used here, but maybe should!). By demanding == 0, I was only going to (catastrophically) extract the bytes if and only if the Handle pointed to nothingness, which was confirmed not to happen at this point. My GetFirstTempo function was perfectly happy doing nothing after it failed its check and the compiler was perfectly happy to let it not return any value. A modern compiler would have cried foul and I would have investigated it.

So why did the result from the main loop (which is in a function outside main but in the main source file) carry the value of result from an inner function in another source file? Mystery. Did an extra weird luck factor put these 2 stack variables at the same place every time? If I declared another variable in the interim, it wouldn't be the same value??

 

[Crutch]

Oh yeah, GetHandleSize returns the actual handle size. Still that’s indeed a weird coincidence. Glad you figured it out!

 

[Mu0n]

Last bit of problem - I'm not sure I understand every seam of the bitwise operations when it's not done on longs as part of a char to short conversion on top of it.

unsigned short result = 0; //wanna make sure bit 15 isn't doing funny business
char *ptr;

HLock(gHandleFileContent);
ptr = *gHandleFileContent; //first byte is 00
result |= *ptr; //result becomes 0x0000, great
result << 8; //result stays 0x0000, great
ptr++; //*ptr is now 0x96, great
result |= *ptr;  //result becomes 0xFF96 ????
HUnlock(gHandleFileContent);

I tried replacing the last line with result += *ptr;, same result. (WHAT?)


edit - solved with a nuclear option at the end:

result = (result & 0xFF00) | (*ptr & 0x00FF);

 

[Crutch]

Your original code was getting a sign extend on *ptr, since 0x96 has the high bit set. So “result |= *ptr” is getting turned into “result |= a short conversion of the signed value contained in *ptr” which would be 0xff96.

I think if you had declared ptr as “unsigned char *” you might have gotten the desired result from your original code. But yes I personally would also have done something more explicit because I don’t like thinking about how these conversions get done at the byte level either!

Btw you know of course that “result << 8” does nothing right? Did you mean “result <<= 8”?

 

[Mu0n]

Your original code was getting a sign extend on *ptr, since 0x96 has the high bit set. So “result |= *ptr” is getting turned into “result |= a short conversion of the signed value contained in *ptr” which would be 0xff96.

I think if you had declared ptr as “unsigned char *” you might have gotten the desired result from your original code. But yes I personally would also have done something more explicit because I don’t like thinking about how these conversions get done at the byte level either!

Btw you know of course that “result << 8” does nothing right? Did you mean “result <<= 8”?

yup yup yup on all counts.
Thanks guys!