It all comes down to countering correct and never attacking, unless you have counter an opponent's punch. As long as you get used to it you can defeat opponents 20 levels higher, something that is next to impossible on death march on a normal combat encounter. But i do understand OP's point. If you try fist fights the same way you do your swordfights, you will get your ass kicked.
From lore perspective it does not make sense. Geralt should knock out these peasants in 1 or 2 punches, but it is a game, and it has to be somewhat challenging on the highest difficulties. You yourself getting a fair amount of damage is OK though. A strong punch is a strong punch. Even if one of us throws a really good punch on Manny Pacquiao, it will hurt him, that's just how it is.
From lore perspective it does not make sense. Geralt should knock out these peasants in 1 or 2 punches, but it is a game, and it has to be somewhat challenging on the highest difficulties. You yourself getting a fair amount of damage is OK though. A strong punch is a strong punch. Even if one of us throws a really good punch on Manny Pacquiao, it will hurt him, that's just how it is.