-
Updated 2023-07-12: Hello, Guest! Welcome back, and be sure to check out this follow-up post about our outage a week or so ago.
Floppy Emu: an SD Card Floppy Emulator
- Thread starter bigmessowires
- Start date
bigmessowires
Well-known member
OK, I think we're in business! I'm going to switch to the ATMEGA1284 to gain some additional RAM buffering space (though not as much as a 1MB external RAM to buffer the whole disk). That, plus some emulator code improvements, should be enough to support most types of writes on most SD cards, I think. The biggest challenge is still emulating initialization of a floppy disk when using a lower-speed SD card. During floppy initialization, the Mac continuously writes both sides of every track without stopping, so the write rate is the highest of any kind of write operation. Today I thought of a couple of ideas that I think will help with initialization emulation, and other cases where an entire side is written in one pass.
The first idea is to special case the handling of empty sectors. During floppy initialization, the Mac writes 1600 sectors very fast. What’s in those sectors? Zeroes. Instead of buffering a 512 byte sector full of zeroes, I could just set a flag that says “this sector is all zeroes”. Using a bitfield, I could buffer an entire disk’s worth of zero sectors using just 200 bytes of RAM. Those sectors could then be saved to the SD card whenever it was convenient, after the floppy initialization was finished. If a read request arrived before all those zero sectors were saved to the card, the emulator could check the flag first to see if an all-zero sector should be synthesized instead of actually loading the sector data from the SD card. This solution is short and simple, though its usefulness is limited to floppy initialization only.
The second idea is to intentionally create an error condition to slow down incoming data, by exploiting some code that measures the size of the gap between the last sector and the first sector on one side of a track. During initialization of a floppy, after the Mac finishes writing the last sector on a side, it immediately switches back to read mode to measure the gap before the next sector, and confirm that the next sector is sector 0.
From examining the ROM disassembly, I discovered that the disk initialization code uses some kind of progress counter that starts with a value of 7. Every successful side written increments the counter by 1. If the gap is the wrong size, the counter is decremented by 1. If the counter value is still greater than 4, it attempts to rewrite the side again, otherwise it aborts with an error.
By intentionally generating a bad gap size after a full side is written, I can force the side to be rewritten. If I also make the emulator smart enough to detect when data written to a sector is identical to what was already there, then it can ignore the second rewrite. That effectively doubles the amount of time available for saving the track data to the SD card, since every side will be written twice by the Mac.
The bad gap size trick can only be done once per side, or else the progress counter will decrease and the initialization process will eventually fail, so it can’t buy an indefinite amount of additional time. It’s also a little risky, because it means the progress counter will never increase above 7, and any 3 other errors occurring during the initialization will cause it to fail.
I did some simple tests of this idea that look promising. By disabling SD saves, I was able to perform floppy initialization to measure its write speed, even though the initialization ultimately failed during the verify phase. In my initial test, it took 34 seconds to complete the write phase when initializing a floppy. After I added emulator code to generate a bad gap after every other side write operation, the time increased to 59 seconds, with no obvious ill effects.
The first idea is to special case the handling of empty sectors. During floppy initialization, the Mac writes 1600 sectors very fast. What’s in those sectors? Zeroes. Instead of buffering a 512 byte sector full of zeroes, I could just set a flag that says “this sector is all zeroes”. Using a bitfield, I could buffer an entire disk’s worth of zero sectors using just 200 bytes of RAM. Those sectors could then be saved to the SD card whenever it was convenient, after the floppy initialization was finished. If a read request arrived before all those zero sectors were saved to the card, the emulator could check the flag first to see if an all-zero sector should be synthesized instead of actually loading the sector data from the SD card. This solution is short and simple, though its usefulness is limited to floppy initialization only.
The second idea is to intentionally create an error condition to slow down incoming data, by exploiting some code that measures the size of the gap between the last sector and the first sector on one side of a track. During initialization of a floppy, after the Mac finishes writing the last sector on a side, it immediately switches back to read mode to measure the gap before the next sector, and confirm that the next sector is sector 0.
From examining the ROM disassembly, I discovered that the disk initialization code uses some kind of progress counter that starts with a value of 7. Every successful side written increments the counter by 1. If the gap is the wrong size, the counter is decremented by 1. If the counter value is still greater than 4, it attempts to rewrite the side again, otherwise it aborts with an error.
By intentionally generating a bad gap size after a full side is written, I can force the side to be rewritten. If I also make the emulator smart enough to detect when data written to a sector is identical to what was already there, then it can ignore the second rewrite. That effectively doubles the amount of time available for saving the track data to the SD card, since every side will be written twice by the Mac.
The bad gap size trick can only be done once per side, or else the progress counter will decrease and the initialization process will eventually fail, so it can’t buy an indefinite amount of additional time. It’s also a little risky, because it means the progress counter will never increase above 7, and any 3 other errors occurring during the initialization will cause it to fail.
I did some simple tests of this idea that look promising. By disabling SD saves, I was able to perform floppy initialization to measure its write speed, even though the initialization ultimately failed during the verify phase. In my initial test, it took 34 seconds to complete the write phase when initializing a floppy. After I added emulator code to generate a bad gap after every other side write operation, the time increased to 59 seconds, with no obvious ill effects.
techknight
Well-known member
I think everything will be fine if you get some RAM to buffer what you need. Too bad the AVR is natively incapable to handle 1MB of RAM. would be nice, because then your read/write access could be as fast as realtime, and when disk input is idle, it gets written to SD card.
bigmessowires
Well-known member
Whew! It took me a long time to do the board layout, but here it is! The board is about 4 x 1.75 inches, or roughly the size of an elongated credit card. The resistors, LEDs, and odd-sized capacitors are all labeled, so any other small rectangular surface-mount parts you see are 0.1 uF decoupling capacitors.
Anyone see any problems or have any suggestions for improvements, before I send this off to be manufactured?
Anyone see any problems or have any suggestions for improvements, before I send this off to be manufactured?
techknight
Well-known member
whats the CPLD for?
bigmessowires
Well-known member
If you look at the photo in this thread's first post, the CPLD replaces the Altera board shown at the left. Its main purpose is to decode the register select signals from the Mac, and handle reading/writing of disk registers, because the AVR isn't fast enough to do it. It also performs serial-to-parallel conversion in both directions, so the AVR can work a byte at a time, without having to worry about the serial data timing requirements.
I'm using the same Xilinx CPLD as this demo board: http://dangerousprototypes.com/2011/03/29/new-prototype-xc9572xl-cpld-development-board/
I thought of one more possible layout improvement- it might be handy to run the 4 unused CPLD pins to a header somewhere, in case I need them later for anything unexpected. I'm not sure there's still space for that though.
I'm using the same Xilinx CPLD as this demo board: http://dangerousprototypes.com/2011/03/29/new-prototype-xc9572xl-cpld-development-board/
I thought of one more possible layout improvement- it might be handy to run the 4 unused CPLD pins to a header somewhere, in case I need them later for anything unexpected. I'm not sure there's still space for that though.
Dennis Nedry
Well-known member
The chances are really good. As far as I am aware, this is all, or almost all of the hardware necessary to emulate an HD20. The biggest problem is cracking the code of the HD20 and figuring out how it works. From what I understand, it uses the same GCR encoding scheme as floppies, so even a lot of your code could probably be used for the HD20.
Trash80toHP_Mini
NIGHT STALKER
I'm in over my head at this level, but I know enough to be amazed at your incredible progress, Big.
Congrats! :approve:
Congrats! :approve:
techknight
Well-known member
74HC244 i believe.
bigmessowires
Well-known member
Thanks! 244 is a 74LVC244, which is acting here as a 5V to 3.3V level converter.
I think something like HD20 behavior is possible in time, though it might be easier to write an all-new driver for it than to reverse engineer the undocumented HD20 behavior. But I'm intentionally not thinking about that stuff yet... I just want to get standard floppy emulation debugged and working on the new hardware.
I think something like HD20 behavior is possible in time, though it might be easier to write an all-new driver for it than to reverse engineer the undocumented HD20 behavior. But I'm intentionally not thinking about that stuff yet... I just want to get standard floppy emulation debugged and working on the new hardware.
Dennis Nedry
Well-known member
I just discovered this thread and I'm quite interested the more I skim around in it.
How are you accessing the SD card, are you using the SPI mode, or are you using one of the other native SD modes? I had understood at one point that SPI is easier yet much slower. These cards are fast enough to benefit from a USB 2.0 reader as opposed to USB 1.1, so if reading/writing to the SD card is a bottleneck even for 800k floppy emulation, maybe there are things that we can look into here to improve the speed instead of adding RAM.
If you are indeed using SPI, and wish to continue using it, it is not too uncommon for microcontrollers to have a built-in SPI subsystem. This will take care of all of the SPI serial stuff in parallel with the code you have running. Basically, you will have an SPI interrupt whenever the SPI subsystem is ready, and in this interrupt, you read or write the next whole byte in a register and exit the interrupt, for example. Then your code goes back about its business as the SPI subsystem sends/receives more SPI data.
edit:
I just looked at the ATMEGA1284 feature summary document and it looks like it does have an SPI subsystem. This is a pretty nice microcontroller, and at 20MHz, I think there is a lot of potential and flexibility to make this quite an awesome, expandable-to-HD-20-via-firmware-update gadget.
It's kinda funny the way things are nowadays, this microcontroller is WAY more powerful than, say, a Mac Plus processor.
Do hit me up if you would like me to try to find ways to make your firmware more efficient with buffering, interrupts and using subsystems and such.
How are you accessing the SD card, are you using the SPI mode, or are you using one of the other native SD modes? I had understood at one point that SPI is easier yet much slower. These cards are fast enough to benefit from a USB 2.0 reader as opposed to USB 1.1, so if reading/writing to the SD card is a bottleneck even for 800k floppy emulation, maybe there are things that we can look into here to improve the speed instead of adding RAM.
If you are indeed using SPI, and wish to continue using it, it is not too uncommon for microcontrollers to have a built-in SPI subsystem. This will take care of all of the SPI serial stuff in parallel with the code you have running. Basically, you will have an SPI interrupt whenever the SPI subsystem is ready, and in this interrupt, you read or write the next whole byte in a register and exit the interrupt, for example. Then your code goes back about its business as the SPI subsystem sends/receives more SPI data.
edit:
I just looked at the ATMEGA1284 feature summary document and it looks like it does have an SPI subsystem. This is a pretty nice microcontroller, and at 20MHz, I think there is a lot of potential and flexibility to make this quite an awesome, expandable-to-HD-20-via-firmware-update gadget.
It's kinda funny the way things are nowadays, this microcontroller is WAY more powerful than, say, a Mac Plus processor.
Do hit me up if you would like me to try to find ways to make your firmware more efficient with buffering, interrupts and using subsystems and such.
bigmessowires
Well-known member
Good thoughts. Yes, it's using SPI to access the SD card. The native SD interface (4-wire) is supposed to be faster, assuming large data transfers to sequential addresses. But it requires money to license, and the protocol is reportedly pretty complex. I don't know of any low-end microcontrollers that support it in hardware. Unfortunately, I don't think it's a realistic option for hobbyists. But I think the SPI interface can be made fast enough to work.
The floppy emulator is using hardware SPI support in the AVR, but it doesn't help so much. There's no buffer, so the mcu has to do work after each byte sent or received. There are only 16 mcu clock cyles between each byte, which isn't enough to do anything else using an interrupt-driven approach. The real benefit of the hardware SPI support is that you don't have to manually toggle the SPI clock and data lines under program control, so SPI transfer speeds can be faster.
SPI is not used for the emulator-to-Mac communication-- that's handled by the CPLD. In theory it might be possible to do it from the mcu with SPI, assuming a mcu with multiple SPI interfaces, but there are some oddities involving synchronization that might make it difficult or impossible. Since I already needed the CPLD for the disk register read/writes, it was easiest to have it handle the Mac-side serial communication too.
If this sort of thing interests you, then you might enjoy reading some more detailed discussions of the Floppy Emu development from my blog: http://www.bigmessowires.com/category/macintosh-floppy-emu/
The floppy emulator is using hardware SPI support in the AVR, but it doesn't help so much. There's no buffer, so the mcu has to do work after each byte sent or received. There are only 16 mcu clock cyles between each byte, which isn't enough to do anything else using an interrupt-driven approach. The real benefit of the hardware SPI support is that you don't have to manually toggle the SPI clock and data lines under program control, so SPI transfer speeds can be faster.
SPI is not used for the emulator-to-Mac communication-- that's handled by the CPLD. In theory it might be possible to do it from the mcu with SPI, assuming a mcu with multiple SPI interfaces, but there are some oddities involving synchronization that might make it difficult or impossible. Since I already needed the CPLD for the disk register read/writes, it was easiest to have it handle the Mac-side serial communication too.
If this sort of thing interests you, then you might enjoy reading some more detailed discussions of the Floppy Emu development from my blog: http://www.bigmessowires.com/category/macintosh-floppy-emu/
I have to wonder if this might be exactly the sort of application the Propeller MCU would be good for. After all, it has eight simultaneously-executing cores which can be separately dedicated to bit-banging their own sets of I/O lines. Dedicate one thread to handling the SPI traffic, another one or two to Mac-side communication, etc. Also... It'd probably require dipping into Assembly rather than Spin but the performance of the Prop might just be just high enough to ditch the CPLD. (I can't say that definitively, but I've seen some multi-mhz sampling speed claims out of the Propeller forums.)
But of course going there would mean scrapping the whole thing and starting over from scratch...
But of course going there would mean scrapping the whole thing and starting over from scratch...
MinerAl
Well-known member
Similar threads
