I just tried mine and didn't notice anything weird. I moved the mouse vertically and it was quite smooth. That was with an Apple made in Malaysia mouse.
@mfinn it might be worth trying to upgrade to the newest firmware. I had flashed your unit with a hybrid firmware that was slightly older in most regards (to maintain serial connectivity for debugging with Windows machines), but supported a broader set of input devices. Note that this hybrid firmware is the same as what I'm running currently.
@saybur would you be able to share the .uf2 file of that latest firmware you compiled that removed the restrictions based on device handler IDs? I would need to setup my Raspberry Pi 400 to recompile from source.