Debugging via QEMU

[cy384]

Somewhat recently, QEMU support has been developed for the Quadra 800. This is great for a few reasons, but one that I think is uniquely useful is the debugging tools you can use with it. I was going to do a nice full tutorial, but sadly been too busy with work and other matters, so I hope this is enough of a start to be useful.

I won't go into the details of compiling/setting QEMU up, but it's not too hard and I think there are pre-built binaries for mac and windows now. Figure out your ROM, disk images, etc. as usual. Also, you'll need gdb with multiarch capabilities, on Ubuntu this is just the gdb-multiarch package.

My QEMU command line arguments are like so:

./qemu-system-m68k -gdb tcp::1234 -S -boot d -L pc-bios -M q800 -m 64 -bios ~/roms/mac/quadra650.rom -drive file=~/roms/mac/qemu-pram.bin,format=raw,if=mtd -device scsi-cd,scsi-id=3,drive=cd1,vendor="MATSHITA",product="CD-ROM CR-8005",ver="1.0k" -drive file=~/roms/mac/disk\ images/MacOS753.cdr,media=cdrom,if=none,id=cd1

Note specifically the addition of -gdb tcp::1234 -S which tells QEMU to not immediately start running, and to listen for a GDB connection on port 1234.

Next, run gdb-multiarch. Use set architecture m68k to set the architecture and target remote localhost:1234 to connect to QEMU. From here, you're ready to do anything you want. Since I'm interested in ROM hacking, I can set a breakpoint to the start of the ROM code like break * 0x4080000a. Then run continue to start execution.

For breakpoints, I'm using Ghidra to disassemble the ROM and find offsets (just add 0x40800000, which is the base address the ROM gets loaded at).

From here, you can do anything in GDB as usual. Single stepping, printing register state, memory values, breaking on writes to specific locations, and lots more. One neat QEMU debugging extra is recording and replaying emulator state, which lets you do "reverse debugging." Take a look at the QEMU and GDB docs for exact details.

In short, this is a very very useful tool especially for early stages of the ROM boot process, where it's hard to communicate any info through other means (I've been using it to mess with memory detection and configuration.)

(pinging @cheesestraws since they mentioned interest)

 

[cheesestraws]

Thanks! That's a really useful starting point.

 

[Crutch]

Awesome I’m really interested in this too. But … is there an iOS version? (In my defense I did try googling this, answer wasn’t obvious.) I am entirely addicted to carrying my vintage Macs everywhere on my iPad nowadays …. Hey look I have 20 free minutes on the train, let’s patch some traps.

 

[mcayland]

Next, run gdb-multiarch. Use set architecture m68k to set the architecture and target remote localhost:1234 to connect to QEMU. From here, you're ready to do anything you want. Since I'm interested in ROM hacking, I can set a breakpoint to the start of the ROM code like break * 0x4080000a. Then run continue to start execution.

For breakpoints, I'm using Ghidra to disassemble the ROM and find offsets (just add 0x40800000, which is the base address the ROM gets loaded at).

From here, you can do anything in GDB as usual. Single stepping, printing register state, memory values, breaking on writes to specific locations, and lots more. One neat QEMU debugging extra is recording and replaying emulator state, which lets you do "reverse debugging." Take a look at the QEMU and GDB docs for exact details.

In short, this is a very very useful tool especially for early stages of the ROM boot process, where it's hard to communicate any info through other means (I've been using it to mess with memory detection and configuration.)

Whilst working on the q800 series for QEMU, I wrote a utility called list2elf that takes the symbols from the MPW ROM map files and uses them to generate a stub ELF file so that you can also have access to the ROM symbols in gdb i.e.

kentang:/# gdb /tmp/Quadra800ROM.elf
GNU gdb 6.4.90-debian
Copyright (C) 2006 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB. Type "show warranty" for details.
This GDB was configured as "m68k-linux-gnu"...(no debugging symbols found)
Using host libthread_db library "/lib/libthread_db.so.1".

(gdb) target remote :1234
Remote debugging using :1234
[New thread 1]
0x4080002a in CRITICAL ()
warning: shared library handler failed to enable breakpoint
(gdb) b DRAWBEEPSCREEN
Breakpoint 1 at 0x40800530
(gdb) c
Continuing.

Breakpoint 1, 0x40800530 in DRAWBEEPSCREEN ()
(gdb) disas $pc,$pc+0x10
Dump of assembler code for function DRAWBEEPSCREEN:
0x40800530 <DRAWBEEPSCREEN+0>: pea %a5@(-4)
0x40800534 <DRAWBEEPSCREEN+4>: macl %fp,%d4,%fp@(-512)&,%a4,%acc1
0x4080053a <DRAWBEEPSCREEN+10>: 0125000
0x4080053c <DRAWBEEPSCREEN+12>: moveal %a5@,%a2
0x4080053e <DRAWBEEPSCREEN+14>: pea %a2@(-108)
0x40800542 <DRAWBEEPSCREEN+18>: macw %a7u,%d6u,%a1@&,%a4,%acc3
0x40800546 <DRAWBEEPSCREEN+22>: orib #102,%d0
End of assembler dump.
(gdb)

You can find out more information and download the stub ELF file from https://github.com/mcayland/qemu-m68k-macos-utils/tree/main/list2elf.

 

[mcayland]

./qemu-system-m68k -gdb tcp::1234 -S -boot d -L pc-bios -M q800 -m 64 -bios ~/roms/mac/quadra650.rom -drive file=~/roms/mac/qemu-pram.bin,format=raw,if=mtd -device scsi-cd,scsi-id=3,drive=cd1,vendor="MATSHITA",product="CD-ROM CR-8005",ver="1.0k" -drive file=~/roms/mac/disk\ images/MacOS753.cdr,media=cdrom,if=none,id=cd1

Note that with the patches merged into QEMU upstream, it is no longer necessary to specify the "vendor", "product", and "ver" properties for the CDROM as those are now set automatically for both HDs and CDROMs by the machine initialisation code. In other words you should be just fine with:

./qemu-system-m68k -gdb tcp::1234 -S -boot d -L pc-bios -M q800 -m 64 -bios ~/roms/mac/quadra650.rom -drive file=~/roms/mac/qemu-pram.bin,format=raw,if=mtd -device scsi-cd,scsi-id=3,drive=cd1 -drive file=~/roms/mac/disk\ images/MacOS753.cdr,media=cdrom,if=none,id=cd1

 

[eharmon]

A bit tangential, but I have also wondered if a BootBug could be emulated with QEMU to allow for interactive debugging that's aware of the OS context running within: http://www.bitsavers.org/pdf/apple/nubus/brigent/bootbug/

But a full gdb session really is more powerful.

 

[cy384]

Whilst working on the q800 series for QEMU, I wrote a utility called list2elf that takes the symbols from the MPW ROM map files and uses them to generate a stub ELF file so that you can also have access to the ROM symbols in gdb i.e.

Nice, I actually wrote a script to load the ROM map files into Ghidra, same idea. Thanks for all your work on QEMU!

 

[mcayland]

A bit tangential, but I have also wondered if a BootBug could be emulated with QEMU to allow for interactive debugging that's aware of the OS context running within: http://www.bitsavers.org/pdf/apple/nubus/brigent/bootbug/

But a full gdb session really is more powerful.

Should be possible without too much difficulty: looks like BootBug is basically a 16450 UART with a declaration ROM. QEMU already has a 16550 which should be usable as a drop-in replacement, so all you'd need to know is the offset from the slot base address where the UART lives and how the Nubus IRQ is wired.

 

[mcayland]

Nice, I actually wrote a script to load the ROM map files into Ghidra, same idea. Thanks for all your work on QEMU!

My pleasure . If you ever find the time to write some of this up with pics, I'd be interested to see some examples as to how you are using gdbstub in tandem with Ghidra as part of your workflow.

 

[Melkhior]

In short, this is a very very useful tool

It is, the Q800 emulation in QEmu was instrumental for me in implementing the Declaration Rom for the *FPGA, and developing/compiling the associated INITs (for acceleration and hdmi-audio support).

In fact, I suspect QEmu has actually been my primary Mac 68k machine for a while

Nice, I actually wrote a script to load the ROM map files into Ghidra, same idea.

Is that publicly available ?

Trying to figure out why I can't seem to access the superslot area through the '040 PDS (on my Q650), having the ability to load a ROM Map for the Q650 in Ghidra would be super useful to see if $E000_0000 is properly mapped as I/O or not (why it shouldn't be is beyond me, but I've reached a stage where I need to check all my assumptions).

Thanks for all your work on QEMU!

Ditto

 

[cy384]

If you ever find the time to write some of this up with pics, I'd be interested to see some examples as to how you are using gdbstub in tandem with Ghidra as part of your workflow.

While I think it might be possible to directly hook ghidra to QEMU, I'm definitely not doing anything so sophisticated. It's more of a manual, side by side situation. I have the ROM loaded in ghidra, my a traps hack, ghidra's automatic disasembly, labels loaded from the ROM maps, and additional manually added labels/comments in areas I'm working on.

Is that publicly available ?

Yes, it's nothing fancy, just inserts a bunch of labels. Here on github: https://github.com/cy384/68k-mac-rom-maps

 

[Arbee]

MAME lets you configure a Q800 or any supported machine with NuBus slots with a BootBug and has an extensive multi-window debugger (Cocoa on macOS, native Win32 on Windows, Qt on Linux).

 

[Melkhior]

Yes, it's nothing fancy, just inserts a bunch of labels. Here on github: https://github.com/cy384/68k-mac-rom-maps

Very useful to bootstrap examination of the ROM and in particular stuff like the MMU setup. Seems on the '040, unlike on the '030, Apple is using the transparent translation registers instead of using pass-through mapping for the I/O areas.
Thanks for sharing this!

 

[buserror]

Does qemu now have a working shared folder with the host system? Last time I tried, it didn't work. I was looking into using it for my Marchintosh Project but gave up as it was quite painful to exchange files...

 

[eharmon]

Does qemu now have a working shared folder with the host system? Last time I tried, it didn't work. I was looking into using it for my Marchintosh Project but gave up as it was quite painful to exchange files...

What OS? The best bet is to just create an AFP share with netatalk, since QEMU networking works.

But on Linux you can mount the drives loopback natively (when the guest OS isn't running -- so less convenient).

Neither are native QEMU sharing, but I've found AFP more stable than Basilisk or SheepShaver shared folders anyway.

 

[buserror]

Is there a 'canned' version of of netatalk that works on modern distros, that still work with the old macs?

 

[robin-fo]

I always get Netatalk through the PiSCSI installer script, even if no PiSCSI is attached to the system

 

[Arbee]

@buserror Netatalk 2.2.10 was released recently and works with 68k Macs and modern Linux distros. Check the Networking topic here.

 

[noglin]

Thanks for sharing this, very, very cool(!!)

My wish is to set a breakpoint in gdb in CFM and see what happens when I double click an app (New World ROM / PowerPC).

I took the ROM of the guest and used tbxi to dump it, and opened ncod_0_CodeFragmentMgr.pef in Ghidra:
CFLOpenContainer XREF[1]: 1000e810(*)
1000ea38 10 00 a1 c4 addr LAB_1000a1c4
1000ea3c 10 00 e8 e8 addr FragCreateContext = 10001c28

I'm not sure where in RAM this LAB_1000a1c4 would be to set the breakpoint.

0x40800000 + <offset of ncod_0_CodeFragmentMgr.pef> + 0xa1c4

Would it be this address? Does NWM/PPC even load the ROM in a fixed location? And how would I get that offset?

I was going to post this question, but then it occurred to me, I can search for it using gdb!

ghidra:
LAB_1000a1c4 XREF[1]: 1000ea38(*)
1000a1c4 39 82 03 08 addi r12,r2,0x308
1000a1c8 81 6c 00 00 lwz r11,0x0(r12)
1000a1cc 2c 0b 00 00 cmpwi r11,0x0
1000a1d0 40 82 00 0c bne LAB_1000a1dc
1000a1d4 38 60 f4 fe li r3,-0xb02
1000a1d8 4e 80 00 20 blr


(gdb) find /b 0x0000000, 0xffffffff, 0x39, 0x82, 0x03, 0x08, 0x81, 0x6c, 0x00, 0x00
(this range can for sure be chosen better, but I don't really know much about how it gets mapped so I patiently waited)

It found two matches:
0xcf1624
0x1f4f1624

I disasm (and made sure to "set endian big") and confirm they are both the same function, I set breakpoint on both:

b *0xcf1624
b *0x1f4f1624

And then I tried to open an app, thinking it would hit the breakpoint (CFLOpenContainer, I presume would be invoked when parsing the container of the PEF?). However, no breakpoint gets triggered.

If anyone has any pointers on what I should differently I'm all ears

This is very very interesting. Thanks so much for sharing again.

 

[joevt]

If anyone has any pointers on what I should differently I'm all ears

You'll probably want to find or create some gdb scripts that can do stuff like MacsBug or Jasik's The Debugger can.
- heap dumps
- resource maps
- windows lists
- processes
- code fragment manager info etc.
- low memory globals

Basically, if you can parse all the known structures of the OS, then most of the pointers and handles in the heaps can be labeled with what points to them - such that only pointers and handles created by an app for non-OS structures will remain unlabeled.
eg. - window@xxxxxxxx->clip region->region handle->region pointer

All the structures should be described in the elliotnunn/supermario source code.

Or you can use MacsBug or Jasik's The Debugger.

===========

Which version of the NewWorld Mac OS ROM file are you looking at? I think it must be "000-11-03 - Mac OS ROM (New World 6.1) 9.1 Universal Update" since that's the only one with the instructions at code section offset A1C4.

IFS=$'\n'
for thefile in $(grep -l --include "ncod_0_CodeFragmentMgr.pef" -R -E "\x39\x82\x03\x08\x81\x6c\x00\x00" .); do
    echo "${thefile/.pef/.dumppef.txt}"
    mpw DumpPEF -annotate -do All -pi u -dialect PPC32,vec -tb -fmt ON "$thefile" > "${thefile/.pef/.dumppef.txt}"
    perl -0777 -nE 'if ( /(.*39820308.*\n.*816C0000.*)/ ) { print $1 . "\n"; exit 0 } else { exit 1 }' "${thefile/.pef/.dumppef.txt}"
done

./2000-11-03 - Mac OS ROM (New World 6.1) 9.1 Universal Update.rom.src/Parcels.src/MacROM.src/Mac68KROM.src/Rsrc/ncod_0_CodeFragmentMgr.dumppef.txt
A264 A1C4          39820308               addi      r12,RTOC,776
A268 A1C8          816C0000               lwz       r11,0x0000(r12)
./2000-12-01 - Mac OS ROM (New World 6.7.1) Mac OS 9.1 installed on Power Mac G4 (Digital Audio).rom.src/Parcels.src/MacROM.src/Mac68KROM.src/Rsrc/ncod_0_CodeFragmentMgr.dumppef.txt
A24C A1AC          39820308               addi      r12,RTOC,776
A250 A1B0          816C0000               lwz       r11,0x0000(r12)
./2001-02-07 - Mac OS ROM (New World 7.5.1) 9.1 iMac 2001.rom.src/Parcels.src/MacROM.src/Mac68KROM.src/Rsrc/ncod_0_CodeFragmentMgr.dumppef.txt
A24C A1AC          39820308               addi      r12,RTOC,776
A250 A1B0          816C0000               lwz       r11,0x0000(r12)
./2001-04-10 - Mac OS ROM (New World 7.8.1) bundled on iBook (Dual USB) (CPU Software 3.5).rom.src/Parcels.src/MacROM.src/Mac68KROM.src/Rsrc/ncod_0_CodeFragmentMgr.dumppef.txt
A24C A1AC          39820308               addi      r12,RTOC,776
A250 A1B0          816C0000               lwz       r11,0x0000(r12)
./2001-04-24 - Mac OS ROM (New World 7.9.1) Mac OS 9.1 bundled on PowerBook G4.rom.src/Parcels.src/MacROM.src/Mac68KROM.src/Rsrc/ncod_0_CodeFragmentMgr.dumppef.txt
A24C A1AC          39820308               addi      r12,RTOC,776
A250 A1B0          816C0000               lwz       r11,0x0000(r12)
./Mac OS ROM (New World 8.0) Mac OS 9.2 Power Mac G4 Install CD.rom.src/Parcels.src/MacROM.src/Mac68KROM.src/Rsrc/ncod_0_CodeFragmentMgr.dumppef.txt
A24C A1AC          39820308               addi      r12,RTOC,776
A250 A1B0          816C0000               lwz       r11,0x0000(r12)

I don't understand this info:

CFLOpenContainer XREF[1]: 1000e810(*) 
1000ea38 10 00 a1 c4 addr LAB_1000a1c4
1000ea3c 10 00 e8 e8 addr FragCreateContext = 10001c28

Does Ghidra know how to work with PEF files? Does it know how to parse the sections list and unpack the pidata?
A1C4 (code section) is definitely CFLOpenContainer with TVector at 0358 (pidata section).
1C28 (code section) is definitely FragCreateContext with TVector at 0208 (pidata section).
So what are addresses e810, ea38, ea3c ?

===========

Maybe try creating an app that imports CFLOpenContainer and output the pointer to that function? Debug this app and see where the function pointer leads. Does the number of gdb find occurrences change while this app is running?