There's a lot of different options for this era of machine. What you choose will depend on your budget, parts availability, and what technology problems you want to give yourself, along with what problems you want to solve.
SATA is a great idea, but I'll caution that
the only compatible SATA card on US eBay right now is being listed for $350 -- If that's fine, then, yay!
If not, then looking for ultra SCSI disks may be your best bet. The only thing you need to do is make sure that the signalling voltage is the same, there were two, "low" and "high" and if I remember right, most stuff that ever hit Macs was "low", but it's worth checking.
The 8600/9600 can definitely cool whatever server disks you put in, and newer ones will probably run a bit cooler, use the two lower bays below the PCI slots since those were meant for hot-fast disks.
Another option is SD card SCSI replacers, I personally use a SCSI2SD v6 in my 8600/300 and recommend it for this type and era of machine. You can get them for about $100 in the US and then just add a reasonably good SD card. I'm using a somewhat high end UHS-1 card in mine, I have a 32-gig card but any size that exists (up to 512GB AFAIK) should work fine.
If you're looking for a server for your vintage Macs, the AWGS9650 might not be the most practical option, it, again, depends on what your goals are. I'm using
ASIP6 on a PowerMac G4 to huff vibes, not because I think it's actually any good at what I'm asking it to do. (It's not.) Netatalk on Linux or Windows 2003 on a Hyper-V/VMware VM would be more efficient and capable. Even Mac OS X 10.4 in VMware or on an Intel Mac will be more capable, often you can do better for less money, and more flexible.
