• Updated 2023-07-12: Hello, Guest! Welcome back, and be sure to check out this follow-up post about our outage a week or so ago.

Multi-threaded network app development on the 68000?

marciot

Well-known member
Hello everyone,

I would like to start a thread (ha!) on multi-threaded programming on the 68000, in particular, how it might relate to MacTCP programming.

When I began developing MiniVNC, I did a bit of research and learned that there were two predominant ways to write MacTCP applications.

The first style appears to have been introduced by Steve Falkenburg in his article "MacTCP Cookbook: Constructing Network-Aware Applications." He provides synchronous high-level routines that make asynchronous calls to MacTCP functions and then does a busy loop to wait for the operation to complete. MacHTTP seems to be a historic app developed entirely using Steve Falkenburg's methods.

The second style is using asynchronous callback routines, which I learned about in the article "Asynchronous Routines on the Macintosh" in Develop magazine, March 1993. This is very efficient, as it eliminates busy-waiting, but it is much harder to do as it relies on callback routines that are called at interrupt time. Callback-based programming makes very simple things—like keeping state, branching, and looping—very hard. The fact that the callbacks are executed at interrupt time makes it even harder, as most Mac toolbox routines cannot safely be called from an interrupt. In practice, this means you need some mechanism for your callback routines to notify the main application loop to do certain operations, which adds to the complexity.

When I started developing MiniVNC, I knew that threads were a modern-day solution to this problem. I came across something very promising, Ari Halberstadt's ThreadLib, but at the time decided that learning to use both MacTCP and a potentially incompatible threading library would be too much of a heavy lift. So I developed MiniVNC entirely using asynchronous callbacks, which turned out to be just as much of a PITA as I had imagined it would be.

But now that I have been developing MiniVNC for two years, and pretty much know all the ins and outs of MacTCP, I want to revisit the question of whether multi-threading could be a good solution for MacTCP apps. What I wished I had was a library for network programming that would allow for a synchronous programming style like Steve Falkenburg's, but that instead of busy-waiting for MacTCP would suspend a thread until data was available. In addition, a thread could specify whether it wanted to be executed from the interrupt or from the main loop, so the developer would not need to implement message passing and other workarounds to accessing toolbox routines. I suspect Ari Halberstadt's work is a great starting point for this, but it would need additional work.

So, before I dig into this, I have a few questions for the community. Does anyone:
  • Know of any similar projects that might already do this?
  • Have experience with Ari Halberstadt's library?
  • Know what is the last version? I've only been able to locate ThreadLib 1.0d4
  • Know of any real-world apps that already use it?
  • Does anyone know of incompatibilities between it and Mac OS?
 

joevt

Well-known member
I've used "GUSI 2 -- A multithreaded POSIX library" in the past. I don't remember what the minimum system version is that it supports.
Threads are cooperative, so you may need to call sched_yield or sched_yield_now if you're not using a call that will yield by itself.

Regarding Asynchronous routines, I wrote a finite state machine engine that works by chaining Asynchronous calls. It handled branching and looping by using case statements in a switch statement like line numbers in BASIC. Each call had an optional timeout and could be cancelled. The call specifies a line number for the next operation and a line number in case of error. It used Metrowerks PowerPlant. It uses LInterruptSafeBroadcaster and LInterruptSafeListener. It was applied to serial port devices but I suppose it could be used for anything that uses asynchronous routines.
 

joevt

Well-known member
What about System 7‘s Thread Manager?
I think these libraries build on top of that. GUSI implements posix threads using Thread Manager. GUSI implements posix sockets (for accept, select, bind, connect, listen, read, write, etc.). You can have different threads working on different connections.
 

marciot

Well-known member
I've used "GUSI 2 -- A multithreaded POSIX library" in the past. I don't remember what the minimum system version is that it supports.
Threads are cooperative, so you may need to call sched_yield or sched_yield_now if you're not using a call that will yield by itself.
Now this is what I am talking about! I gotta find this and take a look at it, as it exactly what I had envisioned.
Regarding Asynchronous routines, I wrote a finite state machine engine that works by chaining Asynchronous calls. It handled branching and looping by using case statements in a switch statement like line numbers in BASIC. Each call had an optional timeout and could be cancelled. The call specifies a line number for the next operation and a line number in case of error. It used Metrowerks PowerPlant. It uses LInterruptSafeBroadcaster and LInterruptSafeListener. It was applied to serial port devices but I suppose it could be used for anything that uses asynchronous routines.
So what advantage does a state machine using switch statements have over a series of small completion routines that are chained together? The callback routines are the states, and the function pointer you pass to the asynchronous routine are the state transitions. It seems to me like having one callback with as switch statement simply adds overhead in the dispatch. I used the small routine technique in MiniVNC.
 

marciot

Well-known member
What about System 7‘s Thread Manager?

Well, I'll let Ari Halberstadt explain why he wrote his Thread Library:

Thread Library implements nonpreemptive multiple thread execution within
a single application. It does not require any extensions, should work
with all Macintosh models (from the Plus on up), and works with
systems 6.0 (tested on 6.0.5) under Finder or MultiFinder, and
system 7.0. Thread Library compiles into a small library of under 3K,
so it won't add much overhead to your application.

Another simple test application compares the speed of Thread Library
with the speed of Apple's Thread Manager. (Thread Library is about
2 to 3 times faster!) Best of all, the source code, entirely in C,
is free.

He goes on:

After the first release on the internet, there was some discussion on
the news group Comp.sys.mac.programmer as to why anyone would bother writing
or using an implementation of threads when Apple has already provided the
Thread Manager. The short answer is that: I wanted to see how hard it would be
to implement threads; Thread Library is compatible with system 6.0;
Thread Library doesn't require the user to install any extensions; Thread
Library is significantly faster than Thread Manager; and Apple charges
$200 to license Thread Manager while the source code for Thread Library
is free, which is especially important to authors of freeware and shareware
applications.

And then he really twists the knife:

If it took me, a single programmer, 6 days to get threads up and
running (and that not even full-time), why did it take Apple many
years and a big fancy extension to get around to implementing
threads? This hack isn't very difficult. I had something sort-of
working the first day, but it took me six days to get it reasonably
stable and to put in all the verbose comments. It would have been
even easier to implement had I had access to proprietary Apple
information. If you look at the Thread Manager extension, it's really
very simple, and it doesn't do much more than what this library does.
(I developed this library prior to examining what the Thread Manager
does and did not try to copy the Thread Manager.)

So, if these comments are to be believed, the real question is why use the Thread Manager? :LOL:
 

marciot

Well-known member
So this is quite fascinating. If you Google "Ari Halberstadt ThreadLib", you get only five hits. The first is this thread (man, Google already indexed it!) and the remaining four are Apple trademark guides that have the following text in them:

MacDNS: Include the following copyright notice on end-user documentation only: "ThreadLib 1.04 © 1994 by Ari Halberstadt."

So, why did Apple itself choose to use Ari's code over its own Thread Manager? :unsure:
 

joevt

Well-known member
So what advantage does a state machine using switch statements have over a series of small completion routines that are chained together? The callback routines are the states, and the function pointer you pass to the asynchronous routine are the state transitions. It seems to me like having one callback with as switch statement simply adds overhead in the dispatch. I used the small routine technique in MiniVNC.
The overhead in the dispatch is tiny compared to the amount of time it takes some device (hard drive, serial port, etc.) to do some work. A switch statement compares up to log2 n values where n is the number of lines. Or it could be O(1) if the line numbers are consecutive.

The point was to keep all the code in a single function in the hope that it would be easier to read or modify.

Here's an example:
C:
const long kTimeoutMisc = 8;				// 8 ms
const long kTimeoutController = 8;			// 8 ms
const long kTimeoutMemoryCard = 16;			// 16 ms
const long kWaitTimeGood = 16;				// 16 ms = 1/60th of a second = 16.6666 ms
const long kWaitTimeMemoryCardWrite = 14;	// 12 ms for sony playstation card, 13 ms for mega memory card
const long kWaitTimeMemoryCardRead = 0;		// 0 ms
const long kWaitTimeBad = 200;				// 200 ms - 2 sec
const long kWaitTimeSelHigh = 0;			// 0 ms
const long kWaitTimeRestartClock = 4;		// 4 ms ?

#define ReturnError( x ) { mPSXParamBlock->transStatus = (x); GotoLineNumber( kErrorEnd ); }

void
CPSXMemoryCard::DoOperation( short inLineNumber )
{
	enum
	{
		kStart,
		kStartFrame,
		kSelectLow,
		kSerialError,

		kWrite,
		kWriteDone,
		kWriteError,

		kRead,
		kReadDone,
		kReadError,

		kErrorEnd,
		kDoneFrame,
		kSelectHigh,
		kWait,
		kCheckResult,
		kEnd
	};

	switch ( inLineNumber )
	{
		LineNumber( kStart )
			GetNextQueuedOperation( mPSXParamBlock );

			mPSXParamBlock->transStatus = PSX_NoData;
			mPSXParamBlock->ioActCount = 0;
			mFrameAddress = mPSXParamBlock->frameAddress;
			if ( mPSXParamBlock->csCode == PSX_Write )
				mWriteBuffer = (PSXMemoryCardPacket*)&mWriteBufferForWrite;
			else
				mWriteBuffer = (PSXMemoryCardPacket*)&mWriteBufferForRead;
			mWriteBuffer->commandHeader.memoryCardNumber = mPSXParamBlock->memoryCardNumber + 0x81;

		LineNumber( kStartFrame )
			mCurrentReadChar = (UInt8*)&mReadBuffer.commandHeader;
			mCurrentWriteChar = (UInt8*)&mWriteBuffer->commandHeader;
			mWriteBuffer->commandHeader.frameAddress = mFrameAddress;
			
			if ( mPSXParamBlock->csCode == PSX_Write )
			{
				mNumBytesLeftWrite = sizeof( mWriteBufferForWrite );

				memcpy( mWriteBufferForWrite.info.dataBlock, mPSXParamBlock->ioBuffer + ( long(mPSXParamBlock->ioActCount) << 7 ), 128 );
				mWriteBufferForWrite.info.XORcode = CalcXORcode( (UInt8*)&mWriteBufferForWrite.commandHeader.frameAddress, 130 );
			}
			else
				mNumBytesLeftWrite = sizeof( mWriteBufferForRead );

			mNumBytesLeftRead = mNumBytesLeftWrite;
			mCurrentWriteChar -= mAdapter->mNumExtraBytes;
			mNumBytesLeftWrite += mAdapter->mNumExtraBytes;

			mNumBytesDoneRead = 0;
			mNumBytesDoneWrite = 0;

			if ( mAdapter->mClearReadBuffer )
			{
				// clear the input buffer because we don’t want to read bytes that were left in there from a previous error
				// this is also necessary if the user attempts to hot unplug a controller

				mAdapter->mClearReadBuffer = false;
				SetCSParam( mMain, Ptr, 0, mAdapter->mSerialEndpoint->GetReceiveBuffer() );
				SetCSParam( mMain, short, 4, mAdapter->mSerialEndpoint->GetReceiveBufferSize() );
				mMain->ControlOperation( mAdapter->mSerialEndpoint->GetInRefNum(), kSERDInputBuffer, kTimeoutMisc, kSelectLow, kSerialError );
			}
			else
				GotoLineNumber( kSelectLow );
			break;

		LineNumber( kSerialError )
			mReturnResult = mLastParamBlock->ioPB.F.cntrlParam.ioResult;
			mPSXParamBlock->transStatus = PSX_SerialError;
			GotoLineNumber( kErrorEnd );

		LineNumber( kSelectLow )
			if ( mAdapter->mTempSelectUsingHSKo )
			{
				mAdapter->mTempSelectUsingHSKo = mAdapter->mSelectUsingHSKo;
				mMain->ControlOperation( mAdapter->mSerialEndpoint->GetOutRefNum(), kSERDNegateDTR - mAdapter->mSelectLoHSKoHi, kTimeoutMisc, kWrite, kSerialError );
				break;
			}

		LineNumber( kWrite )
			if ( mReadIsPending == false )
				if ( mAdapter->mUsePendingRead )
					if ( mNumBytesLeftRead > 0 )
					{
						mReadIsPending = true;
						if ( mAdapter->mReadBurst )
							mPending->ReadOperation( mAdapter->mSerialEndpoint->GetInRefNum(), mNumBytesLeftRead, mCurrentReadChar, kTimeoutMemoryCard, kReadDone, kReadError );
						else
						{
							short numBytesToRead = 1 + ( mAdapter->mAck && mNumBytesLeftRead > 1 );
							mPending->ReadOperation( mAdapter->mSerialEndpoint->GetInRefNum(), numBytesToRead, mCurrentReadChar, kTimeoutMemoryCard, kReadDone, kReadError );
						}
					}

			if ( mNumBytesLeftWrite > 0 )
			{
				if ( mAdapter->mWriteBurst )
					mMain->WriteOperation( mAdapter->mSerialEndpoint->GetOutRefNum(), mNumBytesLeftWrite, mCurrentWriteChar, kTimeoutMemoryCard, kWriteDone, kWriteError );
				else
					mMain->WriteOperation( mAdapter->mSerialEndpoint->GetOutRefNum(), 1, mCurrentWriteChar, kTimeoutMemoryCard, kWriteDone, kWriteError );
			}
			else
				GotoLineNumber( kRead );
			break;

		LineNumber( kWriteDone )
			mNumBytesDoneWrite += mLastParamBlock->ioPB.F.ioParam.ioActCount;
			mNumBytesLeftWrite -= mLastParamBlock->ioPB.F.ioParam.ioActCount;
			mCurrentWriteChar += mLastParamBlock->ioPB.F.ioParam.ioActCount;
			if ( mAdapter->mReadBurst && mNumBytesLeftWrite > 0 )
				GotoLineNumber( kWrite ); // write some more stuff so mReadBurst can get all it needs to read
			GotoLineNumber( kRead );

		LineNumber( kWriteError )
			if ( ( mReturnResult = mLastParamBlock->ioPB.F.ioParam.ioResult ) == abortErr )
				mPSXParamBlock->transStatus = PSX_AdapterError;
			else
				mPSXParamBlock->transStatus = PSX_SerialError;
			GotoLineNumber( kErrorEnd );

		LineNumber( kRead )
			if ( mAdapter->mUsePendingRead )
			{
				if ( mReadIsPending )
					mPending->SetupAsyncTimeout();
			}
			else
			{
				if ( mAdapter->mReadBurst )
					mMain->ReadOperation( mAdapter->mSerialEndpoint->GetInRefNum(), mNumBytesLeftRead, mCurrentReadChar, kTimeoutMemoryCard, kReadDone, kReadError );
				else
				{
					short numBytesToRead = 1 + ( mAdapter->mAck && mNumBytesLeftRead > 1 );
					mMain->ReadOperation( mAdapter->mSerialEndpoint->GetInRefNum(), numBytesToRead, mCurrentReadChar, kTimeoutMemoryCard, kReadDone, kReadError );
				}
			}
			break;

		LineNumber( kReadError )
			mReturnResult = mLastParamBlock->ioPB.F.ioParam.ioResult;
			if ( mAdapter->mReadBurst )
				mPSXParamBlock->transStatus = PSX_AdapterError;
			else if ( mReturnResult != abortErr )
				mPSXParamBlock->transStatus = PSX_SerialError;
			else if ( mLastParamBlock->ioPB.F.ioParam.ioActCount == 1 )
				mPSXParamBlock->transStatus = PSX_NoDevice; // missing ack byte probably means no memory card
			else
				mPSXParamBlock->transStatus = PSX_AdapterError;
			GotoLineNumber( kErrorEnd );

		LineNumber( kReadDone )
			mReadIsPending = false;
			if ( mAdapter->mAck )
				if ( mNumBytesLeftRead > 1 )
					if ( mLastParamBlock->ioPB.F.ioParam.ioActCount > 0 )
						mLastParamBlock->ioPB.F.ioParam.ioActCount--; // remove ack byte

			short mNumBytesDoneBefore = mNumBytesDoneRead;
			mNumBytesDoneRead += mLastParamBlock->ioPB.F.ioParam.ioActCount;
			mNumBytesLeftRead -= mLastParamBlock->ioPB.F.ioParam.ioActCount;

			If_DoneByte( 3 )
				if ( mReadBuffer.commandHeader.thirdByte_Z != 'Z' )
					ReturnError( PSX_BadThird )
			If_DoneByte( 4 )
				if ( mReadBuffer.commandHeader.fourthByte_5D != ']' )
					ReturnError( PSX_BadFourth )

			if ( mPSXParamBlock->csCode == PSX_Write )
			{
				If_DoneByte( 136 )
					if ( mReadBuffer.info.write.endMark_5C != '\\' )
						ReturnError( PSX_BadEndMark1 )
				If_DoneByte( 137 )
					if ( mReadBuffer.info.write.endMark_5D != ']' )
						ReturnError( PSX_BadEndMark2 )
			}
			else
			{
				If_DoneByte( 7 )
					if ( mReadBuffer.info.read.commandAck_5C != '\\' )
						ReturnError( PSX_BadCommandAck )
				If_DoneByte( 8 )
					if ( mReadBuffer.info.read.dataHeader_5D != ']' )
						ReturnError( PSX_BadDataHeader )
				If_DoneByte( 10 )
					if ( mReadBuffer.info.read.dataAddress != mFrameAddress )
						ReturnError( PSX_BadAddr )
				If_DoneByte( 139 )
					if ( mReadBuffer.info.read.XORcode != CalcXORcode( (UInt8*)&mReadBuffer.info.read.dataAddress, 130 ) )
						ReturnError( PSX_BadXOR )
			}

			if ( mNumBytesLeftRead <= 0 )
			{
				UInt8 theChar;
				if ( mPSXParamBlock->csCode == PSX_Write )
					theChar = mReadBuffer.info.write.endFlag_G;
				else
					theChar = mReadBuffer.info.read.endFlag_G;

				if ( theChar == 'G' )
				{
					mFrameAddress++;
					GotoLineNumber( kDoneFrame );
				}
				else if ( theChar == 'N' )
					ReturnError( PSX_IOError )
				else
					ReturnError( PSX_BadEndFlag )
			}

			if ( mAdapter->mAck )
				if ( mCurrentReadChar[1] != 0xFF )
					ReturnError( PSX_BadAck )

			mCurrentReadChar += mLastParamBlock->ioPB.F.ioParam.ioActCount;
			GotoLineNumber( kWrite );

		LineNumber( kErrorEnd )
			mAdapter->mClearReadBuffer = true;

			if ( mAdapter->mUseClockAckThrottle )
			{
				mAdapter->mTempSelectUsingHSKo = true; // restart clock by making HSKo high - forcing clock to be enabled long enough to clear write buffer which will then make SEL high
				mAdapter->mTempWaitTimeSelHigh = kWaitTimeRestartClock;
			}

		LineNumber( kDoneFrame )
		LineNumber( kSelectHigh )
			// for some unknown reason if there is nothing connected to the serial port (gPort on B&W G3), this will fail after 10 or 11 times but works the second time after that
			// It is really noticable when kWaitTimeBad is set to about 200
			if ( mAdapter->mTempSelectUsingHSKo )
			{
				mMain->ControlOperation( mAdapter->mSerialEndpoint->GetOutRefNum(), kSERDAssertDTR + mAdapter->mSelectLoHSKoHi, kTimeoutMisc, kWait, kSelectHigh );
				break;
			}

		LineNumber( kWait )
			if ( mAdapter->mTempWaitTimeSelHigh )
			{
				long theWait = mAdapter->mTempWaitTimeSelHigh;
				mAdapter->mTempWaitTimeSelHigh = kWaitTimeSelHigh;

				mMain->WaitOperation( theWait, kCheckResult );
				break;
			}

		LineNumber( kCheckResult )
			if ( mPSXParamBlock->csCode == PSX_Write )
			{
				memcpy( (UInt8*)mPSXParamBlock->extraData, &mReadBuffer.commandHeader, 6 );
				memcpy( (UInt8*)mPSXParamBlock->extraData + 6, &mReadBuffer.info.write.XORcode, 4 );
			}
			else
			{
				memcpy( (UInt8*)mPSXParamBlock->extraData, &mReadBuffer.commandHeader, 10 );
				memcpy( (UInt8*)mPSXParamBlock->extraData + 10, &mReadBuffer.info.read.XORcode, 2 );

				memcpy( mPSXParamBlock->ioBuffer + ( long(mPSXParamBlock->ioActCount) << 7 ), mReadBuffer.info.read.dataBlock, 128 );
			}

		
			if ( mPSXParamBlock->transStatus == PSX_NoData )
			{
				mPSXParamBlock->ioActCount += 1;
				if ( mPSXParamBlock->ioActCount >= mPSXParamBlock->ioReqCount )
					mPSXParamBlock->transStatus = PSX_GotData;
				else
				{
					mMain->RescheduleOperation( mPSXParamBlock->csCode == PSX_Write ? kWaitTimeMemoryCardWrite : kWaitTimeMemoryCardRead, kStartFrame, false );
					break;
				}
			}
			else
				if ( mReturnResult == noErr )
					mReturnResult = -1;

		LineNumber( kEnd )
			mMain->EndQueuedOperation( mPSXParamBlock->csCode == PSX_Write ? kWaitTimeMemoryCardWrite : kWaitTimeMemoryCardRead, kStart, false );
			break;

	} // switch
}


CPSXMemoryCard is a subclass of CAsyncOperation and LInterruptSafeBroadcaster.

CAsyncOperation defines methods for doing Control, Read, Write, Wait, Reschedule, Jump, and EndQueued operations. It also defines these macros to be used in the DoOperation method:
C:
#define GotoLineNumber( x ) goto x
#define LineNumber( x ) case x: x:
CAsyncOperation adds timeouts for async APIs that don't have a timeout. It handles the errors either from invoking the async call or from the async callback. It handles housekeeping of the paramblocks. mPSXParamBlock is a paramblock representing the entire operation which performs a series of smaller operations (Read, Write, Control, etc.) that have their own paramblocks.
 
Top