Weird Facts about the Menu Manager

[Crutch]

I am writing an INIT that allows hiding/showing the Balloon Help/Apple Guide menu at any time. (Helium lets you do this, but requires a restart. Mine will work without even actually installing the INIT.) It will be released very soon. This involved learning a lot about the undocumented weedy details of the Menu Manager. (Hiding the Help Menu is super easy. Re-showing the correct help menu for any given application - they might all be different - on demand is vastly trickier.) Sharing here some of what I’ve learned!

• The SystemMenuList, an undocumented lomem global at 0x286, is a handle to a MenuList structure (also undocumented but see below) containing "default copies" of the "system menus" (help, application menu, script menu if applicable, etc.). The SystemMenuList is used only to store default copies; only the regular MenuList is ever drawn/used directly. The regular MenuList ends up (see below) with handles to the system menus after the regular menus (followed by the app's hierarchical menus at the end). If you call InsertMenu with a beforeID of 0 to put your menu at the end of the list, the OS patches out InsertMenu to instead insert your menu before the first system menu. Of course, each application has its own MenuList.
• The "calc" routine in the standard MBDF automaticallty adds the system menus to the (regular) MenuList if (1) there is an Apple menu in the MenuList and (2) there aren't any system menus yet (identified as such by large negative menu IDs).
• GetMenuBar removes any system menus from the returned handle. SetMenuBar reinstates the "default" versions from the SystemMenuList. Weirdly, that means that if an app calls _HMGetMenuHandle and then uses _AppendMenu to add custom items to the help menu, and then calls GetMenuBar then SetMenuBar, the custom help items will be gone.
• GetNewMBar doesn't just read in a resource -- instead, it calls GetMenuBar to save the current menubar, then creates a new MenuList, populates it by calling InsertMenu repeatedly, then returns a handle to that MenuList before using SetMenuBar to restore things. (So any app that calls GetNewMBar could trip an InsertMenu patch before any menu bar is actually visible.)
• ClearMenuBar nukes the whole MenuList including the hierarchical portion. (THINK Reference incorrectly states that it simply visually erases the menubar on the screen -- this is wrong.)
• InsertMenu will always first (1) insert the given menu into the hier portion of the app's menulist if beforeID == hierMenu == -1. Otherwise, (2) it checks for a large negative menuID and, if it finds one, sticks the given menu (e.g. Help menu) into the SystemMenuList -- else (3) into the regular portion of the MenuList. It’s impossible to use InsertMenu to directly insert a system menu (as indicated by a large negative ID) into the regular MenuList. (SetMenuBar accomplishes this by using Munger.)
• DeleteMenu will delete the given menu from anywhere in the app's MenuList (regular or hierarchical portion), but won't touch the SystemMenuList (but see below). It will do nothing if given a bad menuID.
• GetMHandle only returns a handle to the menu if found in the app's MenuList (regular or hierarchical), never the SystemMenuList. It will nicely return NULL if given a bad ID. However, we can force GetMHandle to get a handle from the SystemMenuList by setting MenuList = SystemMenuList first (then restoring it). (This also works for DeleteMenu.)
• For the Help menu in particular, by default the MenuList just ends up with a handle to the system help menu in the SystemMenuList. However, if an app calls the _Pack14 routine HMGetHelpMenuHandle(), it will always get a handle to a new, app-specific copy of the Help menu, because the Help Manager assumes the app is asking for the handle so it can append its own items. At the same time, HMGetHelpMenuHandle() appends a divider (dashed line) to the bottom of the new Help menu. This is done by checking to see if the app's help menu handle == the one in the system menu list. If not, presuming the app already has its own copy, the app's handle is just returned. Otherwise (the handles are currently equal), a copy of the system help menu is made (with HandToHand) and munged into the MenuList (with Munger ... not InsertMenu).
• Apple Guide patches the Menu Manager, including a tail patch to _SetMenuBar (which might replace whatever help menu _SetMenuBar re-inserts with something else) and a patch to _InsertMenu.
• Desk Accessories and certain apps (BBEdit) get a dashed line at the bottom of their Help menus when Apple Guide is running. This is due to someone (presumably the Apple Guide patches) calling _HMGetHelpMenuHandle extraneously.

Saving here for posterity in case this info is ever interesting/useful to anyone besides me!

The MenuList itself is a handle like this:

typedef struct {
    MenuHandle        menuHdl;
    short            leftEdge;
} MenuPair;

typedef struct {
    // offset from start of *this* HierMenuListStruct to the
    // MenuPair for the last hier menu (if 0, no hier menus):
    unsigned short    lastHierMenu;
   
    long            iDunnoWhatThisIs;
    MenuPair        hierMenuPairs[];
} HierMenuListStruct, *HierMenuListPtr;

typedef struct {
    // lastMenu is the offset from start of this struct to the MenuPair
    // for the last menu in the menulist (if 0, there are no menus)
    unsigned short    lastMenu;
   
    short            lastRight;
    short            notUsed;
    MenuPair        menuPairs[];
   
    // ... followed by the hier portion of the menu list, which is an
    // embedded HierMenuListStruct (see above) and starts at an offset
    // of lastMenu + 6 from beginning of the MenuListStruct

} MenuListStruct, *MenuListPtr, **MenuListHandle;

 

[cheesestraws]

I love this stuff. Thanks!