A/UX needs a Memory Management Unit, something which only the Mac II and successive higher-end Mac models had.
To be pedantic, the Mac II technically didn't have an MMU out of the box. (It had a socket for one, or you could get one by adding a 68030 upgrade.)
The bigger issue with running A/UX on BasiliskII or Mini vMac (with its experimental MacII support) is that even if one enabled MMU emulation for the 68040 core (Which would probably be "doable" with BasiliskII at least, as its CPU emulation is lifted from UAE and a patch *did* exist for adding MMU support good enough to run Amiga Linux, and MMU support also exists in the Atari ST emulator ARAnyM.) the Mac emulators don't actually emulate most Macintosh *hardware*. For instance, disks are emulated by patching the Mac ROM floppy driver, not by actually implementing the genuine SCSI hardware of a real Mac. The screen hardware, sound, network driver, SCSI manager (which isn't used for disk images), and most everything else is similarly "faked".
For the most part A/UX doesn't call the Apple ROM drivers, and instead hits the hardware directly. So... faceplant.
The only Mac emulator I know of that emulates the hardware more explicitly is MESS, which at least has experimental support for the NCR SCSI controller and several other literal devices. However, the support for the II-series machines is seriously bleeding edge and experimental. (And as anyone who's used it can tell you, using MESS can be a mess even for the "well supported" systems.)