Go with what QuadSix50 said: NetBSD, OpenBSD, or Linux. There are good reasons for that, but I think that the main one that you're going to run into are a limited number of daemons for the older operating systems.
I also wouldn't waste your time trying to get NextSTEP running on a Mac. It would be a difficult multistep process that you would have to do entirely manually, probably develop your own tools, and it is still bound to end in failure. It would go something like this: you would need a bootloader, to get the kernel into memory. If you're lucky, all you need to do is figure out which memory address is has to be loaded into memory at and where the entry point is. Then you jump to that address. I'm almost willing to bet that it won't be that easy though, since the firmware usually does a bit of work before handing control over to a kernel. I have no idea what that would entail. Even if you successfully get the kernel loaded, you have a problem. You're running 100% blind. Even if the Next hardware used the exact same keyboard, display, and serial controllers as a Mac, chances are that they would be found at different addresses. But chances are they won't use the same controllers. After all, all of that is the hardware designers discretion. Even if you could swing all of that, you would still need some way to get a filesystem into place, with all of the files installed in their proper location. For that you would probably have to develop your own tools. But that probably wouldn't even matter anyway, since there is about a zero percent chance of the SCSI controller working without serious patching to the kernel.
To give you a rough idea of how hard this would be, consider this: a friend of mine, a NeXT nut with his own NeXT workstation, tried to load the Intel version on an Intel system. It failed because it didn't recognise the SCSI controller. And the Intel version wasn't designed specifically for NeXT hardware.
I mean, it can probably be done. But in order to get it done you would have to be a genius on an order that would make someone with the combined intelligences of Torvalds and Stallman look like a dimwit; as well as enough patience and stamina to make Ghandi look rash and undetermined. But yeah, I'm sure it can be done. }