When that part of the setup came up I chose 1280x1024, 1024x768, 800x600 and 640x480. The 1024x768 and lower resolutions were already selected.
I know the monitor it is connected to can run at 1280x1024 because other computers I have used with it work.
When I try to change the resolution in Debian I get the options of 800x600 and 640x480.
I'm not asking what you tried to set up as options. When the machine initially boots to Mac OS, before starting Linux, what is the resolution? In my (albeit limited) experience, you will get whatever the Mac OS is set to.
If you can actually switch between the 800 and 640 in Linux, that would prove that I don't know what I'm talking about, and you really do have some kind of problem with the conf files or something.