G5s (and most G4s) are in a really interesting place. They're not really "retro" -- they're definitely somewhat proprietary, and compared to the Macs we've got today, they're odd, finnicky old beasts with limited hardware support, etc. I would be zero surprised in a few years to find that people are comparing the likes of PowerMac and iMac G5s to SiliconGraphics and Sun MIPS/SPARC workstations. Useful in their day, but didn't keep very well and had a fairly limited shelf life after their vendors stopped actively supporting them. (Maybe HP/HPPA or DecPaq/Alpha is a better example than Sun here, because I hear one or more of the BSDs has very active sun/sparc support.)
All of that having been said, if I had a G5, I'd probably use it for much of what I would have done the day it was new in 2003 -- as little Internet as possible, minus maybe an IRC client or an ssh/telnet session, and period-appropriate productivity and graphics or media apps.
At this point, programming for those older versions of 10.x would only really be useful as an exercise in patience, and if you're really intent on porting. (ClassicHasClass may be the best resource for what to do to get started with programming for old versions of Mac OS X. I know that 10.3/10.4 had xcode in the box, starting with 10.5 it may or may not have been on the install CD, but I'm also sure xcode is available on Apple's web site.)
Other than that, I would keep it around. If it's not broken now, it won't be any time soon. The bad caps issue really affected the eMac and one or two models of the iMac G5 the most significantly, so a G5 would be worth cracking open every few years to check on, but probably won't be wildly leaky.
The unfortunate thing is that unless you're running a pretty specific floating-point heavy workload, a G4 or almost any modern Intel computer (Prescott Pentium 4 and newer, Atom, maybe even Pentium M) will outrun it, numbers-wise, and finding updated software for Mac OS X 10.6/10.7 on Intel and Windows/Linux on x86 is way more easy than it is on Mac OS X/PPC. Additionally, a machine with an atom in it will take a tenth (probably less) the electricity that a G5 does, while still being faster and having more/faster interconnects. (There are atom boards available with six SATA-3G/6G ports, and multiple PCI/PCIe ports for additional HBAs, if you're really interested in having a server with any big amount of storage capacity.)
That having been said. The G5 is fairly recent and because it's a Mac, a whole bunch of commercial software was available for it in its day. What can't you do with a G5? Technically, a G5 could do everything I do on my current computer. It would be slower and I would arguably be wasting electricity, but it could be fun, and ram/disks are so cheap, if you're running less than 4 or so gigs of ram and there's any indication your disks might not be performing well, the cost of more ram and new disks would make it worthwhile. (I suspect with 4 gigs of ram, and a new disk or two, 10.4 or 10.5 would run pretty well on a G5, and may even be useful, but it depends on your workload and patience.)
Another thought: Is there anything you want to do with a G5? There are a few of us around who might remember what app people would've grabbed back when G5s were new to accomplish that task.