If I had to hazard a guess it would be that your CD is corrupt/slash/your CD-ROM drive doesn't like it. I had A/UX installed on a Quadra 650 ages ago and it was *mostly* straightforward to get going, but a significant problem I ran into was incompatibilities with modern CD-R media and the old CD-ROM drive. (Which was actually newer than the Q650; it was a pull from a dead Power Mac 7200.) I vaguely recall what finally worked was an oooold piece of 2x CD-RW media; I burned at least two other disks that failed partially through.
(This same issue crops up a lot with older computers of all sorts. I basically gave up on using the CD-ROM drives in older SUN boxes back when I used to install NetBSD on them and would netboot instead, and I ended up doing the same when I wanted to put Debian linux on a late 1997-vintage Dolch portable, although in that case the media issues might have been less severe than the BIOS problems.)
On the topic of monitor resolutions, I did also find I had some problems with A/UX when I had my monitor adapter set for "Multisync". It really preferred to have the sense pins set up for one of the standard Apple fixed frequency resolutions. I had one of those "dial-a-resolution" adapters so just spinning the wheel to the Apple 21" setting worked. Note that A/UX only supports 256 color depth (or less) on the Quadra/Centris 650, it may well get confused if you try to start/install it in the 16 bit color mode.