I've got the driver working surprisingly reliably now, latest version here
I can copy applications (testing using ResEdit), open the applications on the HTTP Disk, do directory listings, copy files to and from, open & edit files on the disk, etc. without issue at this point.
The macsbug debugging calls have been left in still, so the previous disclaimers on that still apply. They're only in error conditions, and I haven't hit any on my test environment (supporting both read & write) recently, so it does seem fairly reliable.
There are 3 parts to this:
1) The driver. When the driver is opened, it takes some parameters such as the address of the server, the name of the server (to work with ghosts), and the file path on the server. It then does a HEAD request to figure out how big the file is, creates a DriveQEl structure, adds it to the drive queue, and posts a notification that the drive has been attached, and returns to the caller. The system processes the posted diskEvt, does some control calls to get the default icon and some stuff, and then goes on with the reads & writes. The driver then translates the reads & writes to HTTP ranged GET and PUT operations. If the file is not writeable, the PUT operations fail with writErr. The driver exists as a DRVR resource within the Control Panel.
2) The INIT/System Extension. This loads the driver on boot. It only loads the driver, doesn't open it, so the driver is basically doing nothing in RAM. This also displays the Control Panel's icon during the boot process. The INIT is just a resource in the Control Panel, not a separate file.
3) The cdev/Control Panel. This is still a pretty rudimentary UI that asks for the hostname & file path of the file to mount as a disk. On open, the Control Panel opens the DNS resolver (finds a resolver resource in MacTCP or TCP/IP control panel, or MacTCP DNR file inside the System Folder). When you hit the "Mount!" button, it attempts to resolve the hostname, open the HTTP Disk device driver (named ".Webdrv") that was loaded by the INIT at boot time.
There is currently no error reporting by the Control Panel, so when it has problems with opening the DNS resolver resource, or resolving the hostname, etc. it just seems like nothing happened. I usually close the control panel, reopen it, and try again. Fixing this is on my TODO list, but my focus has been getting the driver working reliably. Now that it seems to be working well, I'll try to focus on getting the Control Panel nice and pretty.
The "known-bad" things that I mentioned the driver was doing before was calling MacTCP synchronously. MacTCP is its self a device driver, which then calls the ethernet (or appletalk, in the case of MacIP, or modem or whatever is backing the connection). So when my disk driver called into another driver synchronously, and that driver then calls into other drivers, it would occasionally deadlock the system. The badness of this is described in this
tech note. So, I had to restructure/rewrite my straightforward MacTCP calls as an asynchronous state machine. Once I did that, the deadlocks disappeared, and I was able to better handle large read/write requests. I've noticed the File Manager during file copies and other operations can request 256KB in a shot, which is both larger than my network buffers, and MacTCP read/write calls only take a 16bit integer for their calls, so the async state machine handles looping of the MacTCP calls to ensure everything gets read/written.