In case anyone else runs into this issue in the future, it turned out to be a faulty LAG IC in my case.
I checked the 68k with a logic analyser and found that it was stuck in this loop for way to long:
This lead me to check the LAG vsync output which was running at 980Khz, rather than the 60hz...