HTTP block device driver

[LOOM]

With a original ROM we would get a chicken and a egg problem, where we can't run the program without MacTCP and the control panel, and we can't load it from anywhere either because the ROM size is too small to include a full system.

One thing I guess you can do with dougg3's ROM is that you can add a system disk with MacTCP on it [1], because the ROM is bigger than normal. That way you can boot the system disk in the ROM, and load a image on a server.

[1] Assuming the machine can read a larger ROM, and the original ROM can be modified to include a full disk with a network protocol/IP stacks and communicating to internal devices on that layer. I'm not familiar with the technical aspects/limits with the ROM modding.

bbraun:

Can you explain about what you said about writes being not reliable? Will the image get corrupted, or not all the data will be written to id[file corruption] or?

 

[bbraun]

As I said, I don't think loading this particular driver in ROM is a good plan. You could put it into a disk image and put the disk image into ROM, and boot the ROM image with my existing ROMDisk driver, and there might even be enough space to do that. The available space for a bootable ROM disk is machine specific. The II/IIx/SE30's system ROM maps in 1MB of ROM address space and uses 256KB of that for the ROM, leaving 768KB to play with. Once the system boots, the OS typically maps in more (dougg3's ROM SIMM holds 2MB). Other models, such as the IIci and Q700, map in more. I'm not sure about the IIfx or IIsi offhand. But when you're using it for booting, you're kind of limited to what the system ROM maps in for you (it is entirely possible to adjust the mappings yourself, but I'm not there yet). Anyway, this is something I intend to play with once I get my hands on one of dougg3's new programmers.

The problem with writes: The image shouldn't necessarily get corrupted. The writes that succeed will be correct. Remember the device driver is operating at a block level though, so if the filesystem required multiple separate block writes to succeed, and only one of them did, the filesystem might be unhappy. If you're concerned about the image getting corrupted in this way, I'd attach it to an emulator and run disk first aid on it. The problem is the system will occasionally hang on a write. Ultimately, this particular problem comes down to MacTCP (and the OT MacTCP compatibility shim, which has subtle differences from actual MacTCP) not really being intended to be called from a device driver, potentially at interrupt time. It also probably doesn't affect UDP, only TCP operations.

 

[tecneeq]

The big topic on everyone's mind seems to be netbooting. Believe me, this is at the front of my thoughts.

Not at all. A reliable readonly release would be perfect for now. After that reliable readwrite. Then all the other ideas. :approve:

 

[bbraun]

This is on my todo, I've just been working on a couple other projects lately. Sorry there's nothing more substantial to report.

 

[tecneeq]

No need to apologize. In fact, it is i who should apologize, my posting sounds a bit greedy i guess. :lol:

 

[LOOM]

bbraun; What development tools do you use for making these 68k programs? And OS?

 

[bbraun]

I use CodeWarrior 10 (not Pro) on System 7.1. It uses the MacTCP interface, which OT emulates (with different behavior from MacTCP), so I keep one 7.1 machine on MacTCP and one with OT.

 

[olePigeon]

Someone should make a control panel that lets you load a series of pictures for the sole purpose of playing a little animation during the loading of extensions. No other point to it. Hehe. :lol:

 

[tt]

In case you're taking feedback, I have a couple suggestions... One is to have the CP remember the last address entered and to have the "read only" checkbox enabled be default. I snooped around in ResEdit to change the default address, I wan't able to figure out how to make "read only" enabled by default. Another feature would be to add a checkbox to enable automatic mounts after startup. I am really digging this CP!

 

[bbraun]

It feels like forever, but I did some work on this again.

HTTP Disk 0.4.9.cpt.hqx

The user visible changes are the control panel remembers the last URL mounted (stored in 'STR ' resource 128), and it is read-only (the checkbox is there, selected, grayed out). I've refactored all of the MacTCP code to be less of a mess, which is part of why it's read-only. I haven't gotten to the write code yet so it's #ifdef'd out. I've also included some cleanup fixes as my understanding of MacOS disk drivers continues to be refined.

 

[bbraun]

HTTP Disk 0.5.cpt.hqx

I've refactored the write support just like the read stuff. I've re-enabled the read-only checkbox, marking it as enabled (read-only) by default. The control panel won't remember the state of the checkbox, it'll default to read-only every time.

One of the other changes I've done here is with HTTP header handling. As mentioned before, this is opening and closing a connection every time a request is made. This includes creating and releasing the MacTCP 'stream' object. A 'stream' can't be released until the connection has been closed by both sides. So if the server hasn't closed their side, but we've closed ours, the stream can't be released, which means no other disk read/write operations can occur. To help things out in this regard, I'm sending the 'Connection: close' HTTP header to tell the server to close the connection after our request, we aren't reusing it. This seems to help out reliability a bunch, at least on servers that honor it, and actually do close the connection promptly.

Anyway, what all this means is, I've had good success with both read and write doing this, at least on 7.6.1 with OpenTransport.

I haven't done any testing with actual MacTCP yet. I also know the Control Panel doesn't work in System 6. System 6 has all the control panels inside the one "Control Panel" desk accessory. System 7 has each control panel in its own window. This means all the UI elements in the dialog item list in System 6 are off by however many other UI elements are in the CP Desk Accessory (usually 2). I just need to update all my dialog item list indices to do the right thing here. Once I do that, this will hopefully work on System 6 with MacTCP as well.

 

[Trash80toHP_Mini]

Very, very cool. Have you tested under 7.0.1 or 7.1 for the Rocket Jockeys?

 

[bbraun]

I've tested it with 7.1 and MacTCP and it doesn't work for a couple reasons, but at least one of them is interesting in why it doesn't work on 7.1 and does on 7.6.1.

When the File Manager on 7.1 does lookups, it does chained async reads. So when it traverses the filesystem, it does a read of one block, and from the completion routine of that read, it inspects the block and issues another read, and so on down the line until it finds what it is looking for. The HTTP Disk driver internally does chained MacTCP operations, so a read of the HTTP Disk issues an async TCPCreate, and from the completion routine of that, issues a TCPOpen, TCPSend, TCPRecv, TCPClose, TCPRelease, and when finally done, calls the original caller's completion routine. That chain gets huge, which means the stack gets huge, runs into the heap and boom hang.

The "solution" is to call the IO's completion routine from IO an interrupt, which is easily accomplished with the Deferred Task Manager. But the catch is, MacTCP processes received packets (indirectly) from the NIC's interrupt handler. The NIC interrupts when it gets a packet, interrupt handler eventually hands the data to MacTCP, etc. So, when the HTTP Disk returns an IO through an interrupt handler in this situation, the File Manager then issues another read operation, which then goes to MacTCP to open a new connection to the server. The TCP SYN goes out, but the SYN ACK is never processed because we're all from an interrupt handler, the TCP open times out, the IO fails, and bad news.

None of this is a problem on 7.6.1 because the File Manager does things differently and the stack never grows too large and there's no problem.

Anyway, knowing is half the battle, right? Now that I know what the problem is, I've got some ideas on what to do to fix it.

 

[tt]

Nice to see these updates. I am getting some instability with Basilisk II with "Read Only" disabled, "unimplemented trap", but my emulated environment might just be acting up. I'll test it out with real hardware soon. I have to say this driver is pretty genius, it is so tiny and works really seamlessly with the OS, sort of like a modern dropbox, but without being annoying. :b&w:

One thing I was wondering from a design of a control panel perspective, is do control panels support copy/paste? I don't know if I ever tried pasting something into a control panel, there was probably was never a need. I tried with HTTP Disk and it didn't work... it's not an issue, I am just curious.

 

[bbraun]

The unimplemented traps are undoubtedly Debugger calls when something with networking went unexpectedly. Is there any chance you could install macsbug, and when it happens type:

stdlog
go

stdlog will dump a bunch of information to a log file on the desktop, and go will resume execution (which may or may not work, but hey, better than an unimplemented trap. maybe.)

What System version & OT/MacTCP are you using? What server are you connecting to? Apache or a service or?

Control Panels can support copy/paste, but they need to do so explicitly. I haven't added that yet.

 

[tt]

I can't get MacBug (6.6.3) to work in Basilisk. Instead of a system bomb, when it is installed the program just completely crashes. Sometimes I also get a hostname error after clicking "Mount". Here is what gets dumped in my terminal, perhaps this info is useful:

Illegal instruction: 4e7a at 03fe789c
Caught SIGSEGV at address 0x74f46163
D0: 001e0100 D1: 000000ff D2: 00000065 D3: 000901e0 
D4: 000001c7 D5: 00000000 D6: 00000000 D7: 00040003 
A0: 00000c30 A1: 03fe6038 A2: 001dfd30 A3: 0009c278 
A4: 0009c278 A5: 03fe6226 A6: 000d0ffa A7: 03fe243c 
USP=00000000 ISP=03fe243c MSP=00000000 VBR=00000000
T=00 S=1 M=0 X=0 N=0 Z=0 V=0 C=0 IMASK=7
FP0: nan FP1: nan FP2: nan FP3: nan 
FP4: nan FP5: nan FP6: nan FP7: nan 
N=0 Z=0 I=0 NAN=0
Segmentation fault

 

[bbraun]

Oh well, looks like Basilisk II doesn't support macsbug. Thanks for trying.

What's your OS/networking config? Are you using OT or MacTCP?

 

[tt]

OS 7.5.5 with OT 1.1.2 according to the TCP/IP control panel.

 

[tt]

I made a graphic for your CP, if you are interested... I found the disk in the HyperCard clipart stack.

 

[olePigeon]

Might I suggest it officially be called Cloud 9?

I was inspired by tt's graphic, and I made a couple Control Panel icons myself.