Jump to content
drybones99

Is there any way to use a BLU Widget disk image with an emulator?

Recommended Posts

I recently figured out how to use BLU (Basic Lisa Utility) to transfer an image of the entire Widget hard disk on my Lisa 2/10 over a null-modem serial cable to my MacBook Pro as what appears to be a raw binary file. The file size is about 10 MB, which seems appropriate, and by looking at it with a hex editor, I can clearly see some of the data in there, including some of raw text data found in the documents that were written in LisaWrite. I wish to be able to use this disk image with a Lisa emulator, such as LisaEm or IDLE, but so far, I have had little luck. IDLE uses .raw files, but when I rename my disk image to "profile.raw", which is the file name that IDLE expects for the virtual hard drive, it fails to boot, with profile error "84". I have tried this with IDLE in both "Lisa 1" and "Lisa 2" modes. According to The Apple Lisa FAQ at "https://lisafaq.sunder.net/single.html#lisafaq-hw-hd-error_84", this error means that the drive is either damaged, improperly connected, or has damaged system files. I doubt that any of these things are true, since the initial process of creating the disk image appeared flawless, and the Widget drive in my Lisa has 0 bad blocks, according to BLU, and has not exhibited any signs of damage or failure. It appears to me that the file format is simply wrong.

 

LisaEm expects the Widget or Profile image to be in .dc42 format, so perhaps I might have more luck there, if I could somehow convert the raw BLU binary file to a .dc42 file. I have already attempted to use the raw file by itself with LisaEm, but after renaming it to "lisaem-profile.dc42", LisaEm simply acts as though there is no virtual disk, asking me instead to create one. This makes sense, given that it expects a .dc42 file, not a raw BLU binary file. Does anyone have suggestions for what I can do to work with this disk image? My ultimate goal is to be able to use this image with any Lisa emulator just as though I were using a real Widget with a real Lisa 2/10. I have also attempted to use a hex editor to try to fit the raw data into a working virtual disk image, to no avail. If it helps, I have attached the raw disk image file obtained from the Lisa with BLU.

 

(NOTE: The original file had no extension, but for some reason, this website adds a long extension consisting of random hex code, along with underscores in place of spaces, and some other slight changes.)

__-_BUILT-IN_PARALLEL_PORT_(10_MB_Widget_Hard_Disk).e32be7a88805689cf1c9393d79ffb35e

Edited by drybones99
Added tags and a note about the attachment file name.

Share this post


Link to post
Share on other sites

The BLU hard drive image format is a bit different to a RAW format, and I suspect this is the source of the problems you're encountering.

 

First off, BLU hard drive images have an extra block at the beginning: it's a copy of the drive's identification block at address $FFFFFF. The data immediately following is block $000000, after that is $000001, then $000002 and so on.

 

This initial $FFFFFF block is not in a format that the Lisa boot ROM expects, which is why you see error 84, or as the ROM source code puts it, "INCORRECT HEADER".

 

So at a minimum you'll need to remove the first 532 bytes from the beginning of your drive image (yes, 532---that's the Widget and ProFile block size---not 512) to get rid of that initial block. But there could be an additional step, which I'll describe in the next post...

Share this post


Link to post
Share on other sites

I removed all of the data up to offset 000010A0 in HxD, which should be the first 532 bytes, since each offset contains 16 hex numbers, and therefore, 16 bits, or 2 bytes. Each offset is in multiples of 10 in HxD. The last portion to be deleted also includes the text string, "NÐERROR". Unfortunately, the "84" error persists in IDLE, and I've given up on trying to use LisaEm for now, simply due to the fact that it needs a .dc42 drive image. I will now wait for your next post for further guidance.

Share this post


Link to post
Share on other sites

Take a look at the BLU manual, specifically "APPENDIX C Hard Disk Image Format". Inside, the following:

 

Quote

Note: This image format is “unnatural” in that BLU is aware of where the tags are in the byte stream (they precede the data for ProFiles and Priam hard disks, and follow the data for Widgets) and of the interleaving performed by ProFile drivers. Other utilities treat the data in an “as-is” manner, which places the tags before the data for ProFiles (after the data for Widgets), and do not de-interleave the blocks of a ProFile.

 

Some background now. When the Lisa OS writes to a ProFile, it applies a 5:1 interleaving to each 32-sector track. As you may know, this means that every "chunk" of 32 blocks gets shuffled around in a manner that makes sequential reads (block $000000, $000001, $000002) faster. This is useful since the ProFile and Lisa take a bit of time to transfer data between drive and computer, during which time the disk is still spinning. By the time the transfer of block $000000's data is finished, the "natural" block $000001 has swung past the head. So, the Lisa OS's shuffling makes it so the next block is treated like block $000001, even though the ProFile calls it something else (and Lisa OS's disk I/O code does too---that's who's doing this secret shuffling, and that's who lies to the rest of the OS and says it's getting block $000001, even when it really gets a different block).

 

Meanwhile, Widget is a more sophisticated device and can apply this kind of interleaving all on its own. This means that the Lisa OS doesn't have to worry about it---and so it doesn't! There's no shuffling or lying as far as the OS is concerned---if it wants to read block $000000 through $000005 in order, it just asks the drive for those blocks, in order.

 

When BLU backs up a ProFile, it undoes the interleaving done by Lisa OS. Suddenly the blocks are unshuffled, and data is in the same order as the Lisa OS thinks it should be.

When BLU backs up a Widget, there's nothing for it to undo.

 

A raw disk image format (like IDLE needs) wants the disk data to be in its true ordering---the same order you'd find it on the disks as if you'd read block $000000, $000001, and so on. That means that if you convert a ProFile BLU image to a raw image, you'll need to redo the interleaving once you've removed the first block (see prior post).

If you convert a Widget BLU image to raw, you shouldn't have to do this. So, just remove the first 532 bytes from the BLU and you should be fine, right?

 

Unfortunately, there may be another catch. It's time to talk about tags... (see next post :lisa:)

Share this post


Link to post
Share on other sites

PS: I'm suspicious of your hex editing job, but thankful for your description of it. You should remove bytes $0000 through $0214 (exclusive), and probably the last bytes of this portion should be the text "Lisa HD Img BLUV0.90".

 

Next post pending...

Edited by stepleton
clarifying that you don't actually remove byte $214.

Share this post


Link to post
Share on other sites

I attempted to edit the raw image with HxD again, this time only selecting to "Lisa HD Img BLUV0.90", which would indeed end at byte $0213, and now, while IDLE still won't boot the system, I get a different error. The error I now receive is error "75", which, according to the Lisa FAQ, indicates that the "system files are damaged". I'm not sure how this could be the case, seeing as it always boots fine on my real Lisa 2, but some progress has at least been made.

Share this post


Link to post
Share on other sites

OK, tags. The Lisa filesystem (as well as some other filesystems that occasionally turn up on 68kmla from time to time) divide blocks into two parts: a 512-byte "data" part and a 20-byte "tag" part, the latter of which holds some filesystem metadata. The annoying thing for Apple parallel port disks is that they are in different places depending on the device. For ProFile, the Lisa OS puts the 20 tag bytes at the beginning of the 532-byte block. For Widget, the OS puts the tag bytes at the end of the block.

 

Now, looking at the BLU manual disk image format appendix again, it seems like we're good, right? After all,

 

Quote

The limited format BLU hard disk image is composed of records of $200 bytes of Data followed by the corresponding block’s Tag bytes

 

So the tag bytes are already in the right place for Widget, right? After the Data? Well, yes, but there may be a catch.

 

I've never used IDLE, but looking at the source code I'm kinda suspicious of how it works as regards Widgets. Right about here we see it deciding that if you have a 10MB image, then you have a Widget. This triggers the use of the $FFFFFF ID block described here, and that's not what a real Widget's ID block looks like---instead, it looks like this (pardon my python):

      'widget': (
          b'Widget-10    '  # NameString
          b'\x00\x01\x00'   # Device.Widget + Widget.Size + Widget.Type
          b'\x1a\x45'       # Firmware_Revision
          b'\x00\x4c\x00'   # Capacity (19,456 blocks, 10 MB)
          b'\x02\x14'       # Bytes_Per_Block (532)
          b'\x02\x02'       # Number_Of_Cylinders (514)
          b'\x02'           # Number_Of_Heads (2)
          b'\x13'           # Number_Of_Sectors (19)
          b'\x00\x00\x4c'   # Number_Of_Possible_Spare_Locations (76)
          b'\x00\x00\x00'   # Number_Of_Spared_Blocks (None!)
          b'\x00\x00\x00'   # Number_Of_Bad_Blocks (None!)
      ) + b'\x00' * (0x200 - 36),  # Pad to $200 bytes.

Note for starters that the real drive type identifier is "Widget-10    " and not "WIDGET       " as IDLE has it. Other things are wrong, too. This so-so forgery may mean that IDLE does not fool the Lisa OS into thinking that you've got a Widget, and that's probably a good thing---if Lisa OS thought you had a Widget, it would try briefly to talk to the drive in Widgety ways, and no Lisa emulator supports this right now, I don't think.

 

So let's assume that Lisa OS on IDLE would think your disk is a ProFile. Well, some things are not right with your raw data, then, since your raw data came from a Widget. For starters, the blocks are not interleaved. Also, the tags are not in the right part of the block for a ProFile. As soon as Lisa OS is booted enough to care about those things, then it'll probably freak out. What it thinks is the tag (from the start of the block) is actually the first 20 bytes of the data portion of the block. And, while it thinks it's loaded block $000001 or whatever, the lack of interleaving in the raw image means that it's loaded a different block. Under those circumstances it's not surprising that it can't boot.

 

What to do? Well, if you're handy with programming, you could try writing a little program to convert your Widget raw image to a ProFile image. I don't know if this will work---I've never tried it. But here's what you'll need to do:

 

First, for all blocks in the image except for the very first one, you'll need to exchange the tag bytes and data bytes. That is, put the last 20 bytes in the block ahead of the first 512 bytes of the block. (You don't do it for block $000000, since that's the bootloader, and it's not quite part of the filesystem per se.)

 

Next, you'll need to shuffle every successive set of 32 blocks in your image according to the interleave map shown here. That is, move block (i) to location (i / 32 + offset[i % 32]).

 

Now this still may not work, for a few reasons:

1. I'm wrong

2. LisaOS does believe IDLE and thinks "WIDGET" means a Widget.

3. LisaOS uses different bootloaders (i.e. block $000000) for Widgets and ProFiles, and even though you've converted your image into a ProFile image, you've still got a Widget bootloader.

 

Anyway, I think that's my advice---and it captures the situation as I see/guess it.

 

As a first PS: you might have seen that the interleave map comes from a file called "blu-to-dc42.c". This is indeed a program that converts BLU images to DC42 images for LisaEm, but as far as getting an image that will boot LisaEm is concerned, I suspect it only works for ProFile images.

 

As a second PS: you might try continuing your quest over at LisaList2. Some of the authors of BLU and LisaEm are known to read that forum :lisa:

Edited by stepleton
Correcting the guidance for shuffling blocks around

Share this post


Link to post
Share on other sites

I am interested in trying to use that "blu-to-dc42.c" program, but I haven't had much experience in compiling other peoples' programs, and CodeBlocks keeps throwing out compiling errors. I have attempted to fix many of them, but I just can't get the program to run. Do you know how I could get this program to run, so that I might be able to attempt to convert my BLU image file?

Share this post


Link to post
Share on other sites

I'd have to know more about your dev setup to give good advice, and even then I'm afraid it wasn't even easy for me to get it to compile on my Ubuntu machine owing to its build mechanism not wanting to compile it for me. I don't really remember how I coerced it into building the tool. (So, I probably can't provide useful advice in this medium, unfortunately, especially if---as it seems---you may be using Windows.)

 

Either way, I'm 90% sure it won't work out-of-the-box for you on any kind of computer. Note here: you can see that the program assumes that the tags are at the end of the blocks, and since LisaEm will emulate a ProFile for any .dc42 file you make, this tag data will effectively be moved to the front of each block. While I'm pretty sure that will be fine for nearly all blocks in your BLU image, I suspect it's bad news for block $000000, whose ordering should probably be preserved. This means that before this code processes block $000000, you'd want to introduce a modification that swaps the ordering of its first 20 bytes and its last 512 bytes.

 

You might try asking on lisalist2.com for more informed advice. Given the state of present-day emulators, I think what you're asking for is this: you want to know if there's a way to convert a BLU image of a Widget with the Office System installed to a ProFile-10 .dc42 image that will work with LisaEm, or a ProFile-10 raw image that will work with IDLE.

 

(Note that there may not be any such thing as "a ProFile-10 raw image that will work with IDLE", since IDLE claims that any 10 MB disk image is a "WIDGET       ", which is not a valid identifier for any Apple parallel hard drive, so far as I know.)

Edited by stepleton
Adjusted the percentage downward :-)

Share this post


Link to post
Share on other sites
1 hour ago, drybones99 said:

Do you know how I could get this program to run, so that I might be able to attempt to convert my BLU image file? 

This isn't just you, it's a right pain to get this to compile.  I have OS X binaries hanging about if they are any use...

Share this post


Link to post
Share on other sites

My main computer runs Windows 10, but I have a MacBook Pro. If you, in fact, have a binary of the blu-to-dc42 conversion program, I would definitely be interested in trying to run it on the one computer of mine that actually does run macOS.

Share this post


Link to post
Share on other sites
5 minutes ago, drybones99 said:

If you, in fact, have a binary of the blu-to-dc42 conversion program, I would definitely be interested in trying to run it on the one computer of mine that actually does run macOS. 

These were compiled on my aged MacBook under OS X 10.11.  I cannot for the life of me remember how I got them to compile except that a lot of swearing was involved.

lisafsh-tools.zip

Share this post


Link to post
Share on other sites

I was able to convert my disk image to a .dc42 file using the provided tools. Unfortunately, I was unable to get it to work with LisaEm. Upon turning on the virtual Lisa 2 in LisaEm, it turns off immediately. Upon trying again, the light stays off, but the virtual Lisa 2 screen turns on with a picture of the hard disk, and it stays frozen on that screen forever. The converted file is attached.

3_-_BUILT-IN_PARALLEL_PORT_(10_MB_Widget_Hard_Disk).dc42

Edited by drybones99
Added converted file.

Share this post


Link to post
Share on other sites

So, if you go here: https://github.com/rayarachelian/lisaem and do a git clone, you'll have nearly the latest and greatest source code, and inside that, there's a scripts directory, in which you'll find a pair of Windows batch files.
I use these to install cygwin with all the required packages to build LisaEm. These batch file expect to be placed on the E:\ drive, (C: is for windows os, D:\ an optical drive and E:\ is the shared folder in virtualbox I use).

You only have do these steps once:

Assuming a 64 bit machine, you'd also need to download the cygwin installer and save it to e:\e:\setup-x86_64.exe and then right click on the batch file, and run as administrator - if you just run the batch file regularly, it will fail to run properly and go in interactive mode.

Once you get cygwin working, you'd copy that lisaem repo inside of cygwin, or just do git clone again, go into the scripts directory and run build-wx3.1.2-cygwin-windows.sh - this will create a copy of wxwidgets 3.1.2 in /usr/local/.
Once done, do export PATH=/usr/local/wx3.1.2-msw/bin:$PATH
I would also do: echo "export PATH=/usr/local/wx3.1.2-msw/bin:$PATH" >>~/.profile to make this permanent for the next time you run cygwin.

Next go back into the top level directory.
Type in ./build.sh clean build
LisaEm will compile, along with the tools (or possibly not if there are bugs)

You could also type in ./build.sh clean build install if you'd like to have it, and the tools install to c:\program files\sunder.net\lisaem

All this is documented in the readme here: https://github.com/rayarachelian/lisaem/blob/master/README.md#special-steps-for-windows

As this is in active development there probably are bugs, and there's no guarantee that the blu-to-dc42 code will work as intended.

And the next time there's an update, you can do git pull, and run just the build command and you're done.

It's not that hard to build except for the initial steps where you have to install Cygwin on windows, or if you're on macos X, installing wxWidgets, for which I provide a script, in the, you guessed it, scripts directory.

If you're trying to build the tools manually outside of the build scripts, well, yes, you're going to run into issues because you'll first need to build libdc42 and then when you compile the tools, you'll need to link that in, etc.

Share this post


Link to post
Share on other sites

Give this is a try, I just compiled it from the current code, but haven't (much) tested it, restore the contents to c:\program files\sunder.net\lisaem
Inside the bin directory should be the tools you're after.

https://lisaem.sunder.net/downloads/lisaem-1.2.7-beta-2020.03.03-win10-x86-64.zip

It's probably a good idea if you run LisaEm to hit control-+ or control-- as it seems the default zoom level is wrong.

I'll take a look at your BLU file when I get a chance.

Currently I'm wresting with mountain lion trying to get a working VM, but ofc there's no brew.sh. if that works and I can get wxWidgets to properly compile and clang to build executables I can use that to build LisaEm for older macs as some of my users are still on 10.9 and 10.10 (and ofc others are on catalina).

There's still a bunch of bugs around the display modes, and mouse location. Some works, others like 2x3y doesn't.

Share this post


Link to post
Share on other sites

I tried to use the beta version of LisaEm you provided, but whenever I attempt to turn on the virtual Lisa 2, I keep getting the error, "cpu68k_makeipclist: But! ipc is null! Stopped at cpu68k.c:cpu68k_makeipclist:1071 with code :501". When trying to use the blu-to-dc42.exe program with my raw BLU image, I keep getting the errror, "WARNING: Error reading BLU header block, fread size did not return 532 bytes block, got 16 blocks!"

Share this post


Link to post
Share on other sites
7 hours ago, sunder said:

If you're trying to build the tools manually outside of the build scripts, well, yes, you're going to run into issues because you'll first need to build libdc42 and then when you compile the tools, you'll need to link that in, etc.

(As a parenthesis: I just want to be clear that when I said building them involved a lot of swearing, that wasn't in any way a reflection on your work or you.  I know I was doing something unsupported in a suboptimal fashion.  I love LisaEm and the tooling and I don't want to sound ungrateful :-))

Share this post


Link to post
Share on other sites

Meh, that ipc list error means there's something wrong with the code on that disk - so likely the conversion isn't working at all. It's indicative of run away code, i.e. a very long string of code (or all zeros) without branches or JMPs since that piece of code works on basic blocks. This sometimes indicates a bug in the cpu core, but more likely it indicates a corrupted disk.

So, I guess the take away is I need a lot more BLU images to test that blu-to-dc42 program with.

As an aside, I used to provide, and will refresh, https://github.com/rayarachelian/lisafsh-tool-libdc42 once I get these bugs fixed.
That repo is meant as a standalone set of disk tools w/o LisaEm.

But don't switch to that yet, as that code is even older, just pointing out that I'll update that after 1.2.7 goes to production incase someone comes back to this thread, say a few years from now and runs into similar issues.

I think the most likely thing is what Tom said, I suspect, but haven't verified that LOS does interleaving for Profiles and possibly not widgets. Again, I don't know this for a fact, so I can't say 100% yet. I'll test some more. At this point this is just a thought, and it doesn't even have enough evidence to be a hypothesis yet.

Thing is I only have one working Widget drive and it has MacWorks on it, so the number of BLU images I have to work with is very limited. If you guys can do me a favor, can you take a few BLU images of widgets and a few of profiles (and clearly mark which are which), I can test the code some more and see if I can get it working, and perhaps get it working with LisaEm too.

But if what Tom says is true, that Widget LOS images have a different bootloader than Profiles, they'll not work on LisaEm until after 1.2.8 or even 2.0. (When I started on 1.2.7 I intended on building widget support, but then Apple went and announced that they'll do away with 32 bit support and then I got stuck on cleaning up the hidpi aware code, so never finished full widget support.)

I've got a few minor cross platform (ie. windows vs macos vs linux) bugs to deal with, and some major ones with the screen handling code before I can publish 1.2.7, and I'd also like to add one or two more features before I ship this beast and move on to 2.0 features, so don't really have cycles to chase too much of this down on actual hardware. But if you guys have the cycles to image a few hard drives, and can save me some time taking a bunch of images, I'll happily work on it. (and I'm not asking you @chesestraws to do all of it, this is a general ask for a favor).

Widgets use a completely different protocol than profiles, though they're backwards compatible with profiles, however, if that boot loader on a widget is really expecting to use the multi-read protocol, it will absolutely not work with current 1.2.7 Lying to the OS and labeling the name of the physical device as "WIDGET" or "WIDGET-10" vs "PROFILE" does not activate this mode, the protocol version does that. This is why SunRem was able to make their own profile compatible third party hard drives. The OS doesn't care about that label.

As an aside you'd think that centering a rectangle with wxWidgets inside a window should be simple, get the width and height of the window, substract the width/height of the rectangle and divide by two and use that as the origin, right?
except that w/GTK on linux and wx3.1.x it returns untrue values (either that or I've got some serious bugs somewhere else). Grrr.

Share this post


Link to post
Share on other sites

When I tried using the .dc42 profile image that I had working with LisaEm 1.2.6 with the new beta version, I got the ipc error again. Furthermore, when I attempted to create a new profile image by installing LOS 3.1 by booting from the first install floppy image, I get "The Tag checksum failed for this disk. It may be corrupted." This is followed by the same ipc error as before and an endless loop of the disk drive sound. The floppy images are in .image format, but they were working just fine in LisaEm 1.2.6, although I get the Tag checksum error there as well, but afterwards, it boots properly instead of giving me the ipc error. I suspect that the disk image may not be the problem here, seeing as I have been unable to get the virtual Lisa 2 to turn on in any manner with the beta LisaEm. If you need a BLU Widget image, I have already uploaded one at the start of this thread, but I'll attach it again to this post. Ignore the long string of numbers in the extension, for this is an artifact of uploading it on this website. The original file has no extension. I have also attached the LOS 3.1 Disk 1 that boots fine in LisaEm 1.2.6, but fails in the new beta version.

__-_BUILT-IN_PARALLEL_PORT_(10_MB_Widget_Hard_Disk).15240174b867249d873369e340911b47

LOS 3.1 1.image

Edited by drybones99

Share this post


Link to post
Share on other sites

Ah ok, thank you for that LOS image. I've downloaded both and will check them.
btw, compressing the BLU image with xz reduces it down to 3.5MB:

-rw-r--r--   1 ray  ray   9.9M Mar  8 21:21 '3_-_BUILT-IN_PARALLEL_PORT_(10_MB_Widget_Hard_Disk).dc42'
-rw-r--r--   1 ray  ray   3.5M Mar  9 19:02 '__-_BUILT-IN_PARALLEL_PORT_(10_MB_Widget_Hard_Disk).blu.xz'
-rw-r--r--   1 ray  ray   410K Mar  9 19:03 'LOS 3.1 1-boots-on-126-but-not-127.image'
-rw-r--r--   1 ray  ray   9.9M Mar  9 19:07 '__-_BUILT-IN_PARALLEL_PORT_(10_MB_Widget_Hard_Disk).blu'



I'll start looking at this today.

Share this post


Link to post
Share on other sites

Good news, I found some bugs in the code, it's now producing an image where the tags are correct, however, bad news, attempting to boot from it throws a 10726 error.

I found a couple of bugs in the converter program, and now, looking with lisafsh-tool, I see that the blocks that have the boot loader on the profile match the converted widget drive. I then tried to repair the Widget image using the LOS install disk #1 and it failed to do so with error 200/663, but not sure what that means.

__-_BUILT-IN_PARALLEL_PORT_(10_MB_Widget_Hard_Disk)_blu.dc42.xz

Edited by sunder
attach disk image

Share this post


Link to post
Share on other sites

Also tried to boot off a good profile image and attach the converted image on a dual port parallel slot (you can do that by changing the preferences, telling LOS there's a 2 port card in one of the slots, and that, say, the upper port is a profile) but the disk isn't recognized.

 

So likely there's some more stuff needed to get this working.

Share this post


Link to post
Share on other sites

Hi, some better news, I tweaked the tool a bit more and now both tags and data is a lot better, I'm able to run the "dir" command in Lisafsh tool and it's also able to recognize the MDDF, likely this is now working, though I don't have time today to test this in LisaEm due to work obligations.

 

I'll test this later and if all is good and I manage to fix other bugs I'll push the code to github, and cut a binary release as well.

 

Hope this helps. :lisa:

__-_BUILT-IN_PARALLEL_PORT_(10_MB_Widget_Hard_Disk)_blu.dc42.xz

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now

×