To do those kinds of visualizations takes an immense amount of processing power.
You may be able to use integer-based math and show an "oscilloscope" like waveform on the screen, but it certainly wont be fast. you basically have to sample the audio at a given frequency, and record the dB levels of each sample in a table for a given time period.
Then this table is painted in a graph on screen. Lots of math work that an FPU would be good at doing, but with a certain degree of error, can be done with integers only.
Spectrum Analyzer visualization FFT? You may be able to use some optimized routines that were targeted for 8-bit MCUs like the Atmel AVR/PIC since they are table based. But again, the sampling takes the most amount of overhead and RAM.
did software exist to do this back then? Yea kinda. probably not on a Mac though. However given the knowledge we have today with doing oscillographs and DFT/FFT on a RISC microcontroller, it can probably be done.
Back in the old days, I used CD Spectrum Pro on windows and it worked ok but it really needed a pentium. a 486 could do it with Goldwave, but only like a 7 band visualizer. 386 was too choppy and had lots of lag.
This topic has kind of peaked my interest because I have been meaning to write a program to do something like this one day on a vintage mac, but my plan was to use external hardware for sampling etc. Would be interesting if it could be pulled off using entirely the original hardware. it would have to be written and compiled in highly optimized C, no way around that.