As Damien mentioned, the difficulty will come from the enemies dealing more damage, thus the accent on the player skill and being able to avoid getting hit. Depending on the damage increase and enemy AI, it could turn out to be hard enough. Also the part with getting less experience on higher difficulties also could affect things.
Having a drinking animation is something that could affect the difficulty, like in the first Witcher, by making you stand still and get pummeled, but it's about balance in the end. Having sufficient difficulty, being fun to play and not being too annoying.
The challange comes from the complexity, not from the healt and hit points of the trash mobs.
Drink potion in combat adds challange because you have to hit the right "time slot" to drink the potion without being hit from the enemies.