Heh...I've always had some level of issue with that every time I played. My guess is that it's simply an engine issue. I've never been able to figure out how to avoid it completely.
For me, I remember using a key ("B", I think...?) to make Geralt walk where I click instead of jog. He would initially walk very slowly. After playing for a while...he'd suddenly start walking at a pretty normal pace. If I reloaded enough times...he'd eventually start walking super-slowly again. (Ultimately, I just gave up on it and jogged everywhere.)
Not sure if that's exactly what you're referring to, but I think it's indicative of the same issue.