I mostly agree with you, and personally I also think that the characters aren't always believable, either. I feel that we're so happy to consider TW a "gritty, realistic franchise" that we ignore just how its fantasy elements are sometimes the stark opposite. People shrunk into statues, a guy who becomes a hedgehog until midnight, Yennefer turning a cart of soldiers into frogs, genies granting wishes, a siren giving up her tail for legs, and that's just off the top of my head. TW has its nasty politics, conniving characters, betrayal, death, anguish, racism, etc. But it also has its fair share of whacky and extravagant fantasy elements, to the point where I believe that saying X fantasy element makes sense but Y doesn't is almost arbitrary and based on a person's personal preferences, no "inner logic of the universe". As a side note, I never got the feeling from the books that magic has some strict structure. I always felt that it did whatever Sapkowski felt was cool for the story.
Edit: I'll admit though that there's some difference between magical capabilities and physical. The latter was almost fairly grounded, and a witcher's main advantage is often described as speed, and mutated metabolism. As in, doing what other humans do, only better. And deflecting an arrow makes you wonder how is that physically possible, unlike magic where you just accept it because... magic. But still, in a world with so many crazy abilities, this is just another one which I accept. To me it's less jarring than Yennefer turning people into frogs, so to each their own, I guess.