It's been a year or so since I fiddled with MESS much, but:
A: That "MessMenu" program is less than useless. Ignore it.
(Or, maybe I'll be kinder: If it actually offers anything over using MESS at the command line I'm too dense to figure out how to use it.)
B: To get the straight-up command line version of MESS going, the drill goes something like this after downloading the .zip file:
1: Create a directory to keep all this mess in: mkdir ~/MESS ; cd ~/MESS
2: unzip ~/Downloads/sdlmess0xxxyz.zip
3: cd sdlmessxxyyzz (whatever the directory it expands out to is)
4: ./mess -createconfig
That last step will make a "mess.ini" file in directory you unpacked MESS into. Load that file into a text editor and take a look at it; the most important things you're concerned with is the first two sections labeled "CORE SEARCH PATH OPTIONS" and "CORE OUTPUT DIRECTORY OPTIONS". The format is "function (whitespace) argument", so for example the default "rompath roms" directive will tell MESS to look in a directory named "roms" inside the current directory (when MESS is started) for the appropriate BIOS rom for the system requested when you start the program. You can either change all these directories to point to directories you create outside the program directory, or just create all the appropriate directories inside the directory you unzipped. (The latter is probably preferable since doing so allows you to "upgrade" MESS just by unzipping a new version and copying your old MESS.ini to it. You could of course even have multiple versions present at once that way.)
(One other thing I would *highly* recommend is down under "VIDEO OPTIONS" change "window" from "0" to "1" and "maximize" from "1" to "0". It's sort of irritating that the default is to grab the whole screen...)
Anyway, once you're all done with that you can dump your ROM sets into the "roms" directory you pointed mess.ini at and fire up an emulation by running "./mess SYSTEMNAME", and you should be pretty much set to follow the various tutorials that are floating around out there.
It is probably worth emphasizing an important point about MESS and "rom sets" if you're new to the program: MESS works a lot like MAME in terms of how it "expects" ROM names (and checksums) to match up with system definitions. If you're looking to use MESS with your own ROM dumps or with modded ROMs it's something you'll have to take into account. You'll find particularly with more complex systems or computers that came in multiple versions that shared BIOS components that if you try starting MAME specifying "computerx" as the system type it will actually want to source a file or two from the BIOS set for "computery" and/or "computerz". You'll definitely need to fiddle with some of the more esoteric command-line options to do "rom replacement" in those cases.
The other thing that's probably worth throwing out there is that unless it's improved *a lot* in the last year MESS' support for machines later than the Mac Plus is highly experimental and a lot doesn't work. (Also it's very difficult to get System versions later than 6.x to work even on macplus, unless they've fixed *that* bug. There is a 7.5.3 install howto on the MESS wiki now... good luck there.) I was able to get System 6 sorta-working on the macse30 emulation, but to do it I had to build a .chd under macplus since SWIM support was completely broken, therefore preventing the reading of floppies in basically any Mac II-based emulation. (Other than the original II, which was broken for other reasons.)
... and... I just noticed something. The versions of MESS on macmess.org are indeed *hella old*. You can try the version from here instead:
http://code.google.com/p/mess-universal/
It's only 4 versions behind, not 14, and at least includes the macII family emulations. (substitute the appropriate commands for expanding the tar file and you can still follow the rest of the instructions above.) Or you could try building it. It's not difficult on Linux, but I imagine most of the hassle on MacOS will be tracking down its dependencies.
Anyway, good luck. I'd just caution that you shouldn't get your expectations up too high. Again, unless it's improved *a lot* I suspect you're going to end up pretty frustrated with the level of buggyness you're going to run into when trying to run the Mac emulations. MESS is a cool idea, but once you get beyond the simple 8 bit computers it at least gives the impression that it's a little less "baked" than its arcade-game running blood brother.
