I've only spent a couple hours mucking with my 660, but I had a similarly frustrating time getting it to reliably switch over.
One of the first things I found was someone had set both CPU and bus speeds to the absolute maximum settings via the dip switches, and even when returning them to stock I found the 500MHz K6-2 would heat up but not work. Installing a replacement CPU brought me to a semi-working state.
It's in my 9600 with some other hardware variables I've yet to isolate, but I quickly noticed in some OS versions it would switch video and begin RAM tests without complaint, and other OSes would either freeze or error out before switching the video source. I don't remember which ones worked, but I don't think 7.6 or 8.6 were very happy with it. 9.2.1 may have worked best... I also faintly remember it wanted to look for the drive image on its native SCSI; wasn't happy with the file on one of the PCI SCSI or ATA drives.
In the time since I managed to get a Blue and White G3 up and running, which I've wanted to test with to see if I have better luck. Orange Micro outlines OPCs 665 and 625 as versions of the 660 and 620 which work specifically with the G3. It could just be a different software bundle, but if it's something in hardware it's possible that's been my hangup.