This has been driving me crazy. I prefer using the controller when driving and casually walking around, and using the mouse+keyboard for action. Starting the game directly from its installation folder, or via GOG or Steam, all yield the same results. The controller will stop working no matter what, at some point or another. I have tried it on a wireless Xbox One controller and a wired Xbox 360 controller. Same thing happens every play session.
However mine doesn't simply stop working when switching between the two. That would be even worse, I suppose. I can switch back and forth fine for some time, but inevitably the controller will randomly stop working eventually. Same issue as those cited here: right bumper, trigger, joystick all stop working.
Restarting the game does fix it, although sometimes I just continue playing and it will randomly start working again. Something must be triggering it in the game that causes it to stop, and then conversely, to start back up again.