Cheesestraws is absolutely right - I do the wash in hot soapy water (with dishwashing liquid) first, scrubbing with a toothbrush, and then after blow-drying, I use IPA to rinse.
I haven’t had any problems with speakers or flybacks - I’m just careful not to touch the speaker when it’s wet. When...