Pan and Tilt fixated on TV

I’ve set up a 2k pan and tilt camera in the lounge and it keeps swivelling around to fixate on the TV. The consequence of this is that the camera doesn’t pick up any new movement as it’s almost looking backwards.

Any suggestions other than moving the camera?