I feel like it annoys me even more when I play with the controller for that reason. On m&kb, one can just move the mouse a bit to cancel the auto camera. But on a controller, when your thumb is not on the stick, you can't move the camera as fast and it becomes a bigger annoyance.