Sounds like going for a Toby clone in FPGA would be the logical first step in testing. Completing that design example with its well documented Hardware/DECLROM/Driver development tools is something for which the gang might enlist for assistance while you're focused on the hardware side?
For the Sun cg6 (a.k.a. GX / TurboGX), I went for cloning as there was a lot of documentation; not only there were some historical open-source drivers in X11, but the chips were made by a third-party (LSI) and the design was cloned by multiple companies; a couple years ago the original Sun documents on the architecture surfaced on bitsavers. And the Forth PROM can be disassembled to the original Forth easily (it's almost a one-to-one conversion between the language and the bytecode, you only lose some clarity on loop structure).
To fully clone Toby (i.e. being able to reuse the binary ROM and driver with no changes), one would need to know the exact addresses and behavior of the various hardware registers, the behavior of the interrupt line, and whether timings are important to the driver - the embedded DDR3 can have spike in latency, e.g. if a request arrives during a refresh cycle; that might be a deal-breaker for an exact clone. Also the link you posted says "up to 7.6.1" and I'm running 8.1

(joking, I could install an older system easily of course, I put a MacSD in the Q650).
It might be easier to create a new ROM/driver based on Toby, but without the constraint of replicating the hardware exactly. It will be needed to go beyond 640x480 anyway.
Parlaying that foundation into a more capable version would be your second iteration and then adapting Toby or jumping into one of the more complex, high resolution examples available like BUG Pickles would be a natural progression?
It would be, and hardware-wise no change would be required; only the FPGA gateware (and maybe CPLD) will need to evolve to support extra functions, that's the beauty of an FPGA. it's also why the software is the big sticking point - you need the same amount of work today to write the software that would have been needed back then, but with only volunteer work and a lot less knowledge floating around. On the other hand, designing and implementing hardware has become extraordinarily easier; VHDL only became a standard the year of the MacII and Xilinx was founded during the Mac 512k era. The speed at which we can design (and have built) SMD PCBs would have been mind-boggling back then - I did my own Mac-to-VGA adapter because it was cheaper to make a couple than buying one second-hand...
Back to NuBusFPGA; should a first version of the board work fully (i.e. NuBus, VGA, HDMI, USB all working at HW level), the only reason to re-spin it would be to change some critical parameters; one might be total cost. Cheaper FPGA and memory directly on the board for instance to be full-custom; there's a
Spartan 7 board Kicad design from an
eevblog forum member that could almost, but not quite, be a good basis; the 196-pins S7 doesn't have quite enough I/O for both DDR2 and talking to NuBus (only 100, DDR2 takes a full bank). But
an Artix-7 in the FTG256 package *might* be doable on 4 layers by an experienced PCB designer (read: not me) and with 170 pins could certainly do the job - and is available in 15/35/50/75/100 flavors; the 15 might be enough for a basic framebuffer while the 100 could likely fit enough hardware to out-perform every vintage board.
Of course the ultimate solution doesn't involve a FPGA at all - after all the
Skywater 130nm mixed-signal process whose
PDK is open source now supports 5V I/O, so it doesn't even need those annoying level shifters.
And we could get Google to pay for it 
. Mostly kidding, as the work needed to get a working chip with support for the required memory would be huge. Mostly...