I think that you almost have to leave the Macintosh out of the picture when you consider the demise of the Apple IIgs. It is all too easy for us to look back and say the Macintosh was the future, because the Macintosh won out. The situation is worse when you consider that most of us are Mac enthusiasts, so we will be even more biased.
Take a look at the following graph (near the bottom of the page):
http://arstechnica.com/articles/culture/total-share.ars/5
You should notice that the Unit sales for the Mac vs. the Apple II line. Even in 1990, the Apple II had a noticeable slice of the pie (though clearly much less than the Macintosh). The Apple IIgs was on Apple's price list until the end of 1992, while the Apple IIe was there a year longer (Wikipedia). This was into the Quadra era. Not bad considering that the updates to these models were very minor.
I think the better question is why did Apple keep the IIgs around for so long. After all, it is pretty clear that they considered the Macintosh to be the future. There are two reasons: the Apple II was a substantial revenue stream and likely had better margins; Apple was also likely concerned about losing the education market (which has a considerable investment in Apple II software and peripherals, without the money to replace it).
Contrary to what was said earlier, the Macintosh was never compatible with Apple II software. The architecture is just too different. The only way to run Apple II software on a Macintosh is to use an emulator or hardware add-on (like the LC IIe card). To my knowledge, there never was a IIgs emulator for 68k Macs. Even if there was, it would never be able to handle copy protected software (which the schools used plenty of). The LC IIe card did allow you to run copy protected software, but it did not run Apple IIgs software.