I could have sworn the G5 xserves had built in video, but I was apparently thinking of the INtel versions, which had ATi X1300, followed by nVidia GT120.
I'm like 99% that Apple let you configure a GPU in the xserves G4 and G5. Not a particularly powerful one, but something that would give you "good enough" desktop functionality on pro apps (in particular, for things like log/capture in final cut, capturing audio in Logic, or controlling renders in 3d apps) so that you could pretty much use the thing as a rackmount workstation.
The Xserve G5 was probably the best incarnation of the G5 ever made; its cooling system was if anything overbuilt and it let the CPUs fly in a way the desktop versions never did.
I'd be interested in seeing numbers showing that an xserve at a particular speed was actually any faster than a Power Mac. The Power Macintoshes generally shipped at higher frequencies (dual 2.7GHz, dual/quad 2.5GHz) and I haven't heard reports of those systems throttling due to heat concerns, which would be interesting to see/hear because Apple made having those cooling channels/chambers a pretty big priority with the Power Macintosh G5, so unless it is actually better for the air to scream by in a small space than for it to scream by in a large space, I would think that the PMac would "win" or that they'd be equals.
Back in the day, I always wanted Apple to build something basically akin to the PowerEdge T610/620/630. Dual socket G5s or Xeons, several disk bays, several expansion slots, into a system that can be ordered as or potentially converted into either a particularly large tower, or a 5u server. I always thought that the potential for a server with its own real built in raid using an enclosure like this was going to be better for the Mac community, and that a a workstation for people who needed even more expansion than the regular G5/Mac Pro it could prove popular.