The Color Classic is an interesting machine. Yours has pretty much the same specs that mine had, only mine had an Apple IIe card in the PDS slot and it ran an English System 7.
The Color Classic is fun to play with but, as you pointed out, slow. It's also incredibly limited. It can only be expanded to 10 MB of RAM. Swapping out the hard drive can be a bugger too. Although, if you can find a 68882 to pop into the FPU slot, you'll see a small speed boost.
Probably best to do the Mystic upgrade. That involves swapping out your Color Classic logic board for an LC 575 logic board. A relatively simple and rewarding endeavour.
The LC 575 uses a 68LC040 that can easily be swapped for a full '040 with builtin FPU. The LC575 is also expandable to 36MB of RAM and runs at a much quicker 33MHz clock speed on a much more capable 68040 processor. The LC575 also has one of the fastest SCSI controllers that Apple has ever shipped in a 68k Macintosh.