It was really difficult until about level 12, but after that it got considerably easier, especially after you get comfortable with the combat. Still pretty hard though, not very forgiving of mistakes, getting hit really hurts, a lot, and Quen spamming comes into play more often than I'd like as a lot of monsters, especially bigger ones, like Trolls or Katakans, or Cyclopses, or Water Hags, etc etc have strange hitboxes where even when you dodge and they don't physically come in contact with you, it'll hit you anyways. Makes it really hard to fight them without Quen, as well timed dodges and in some cases rolls, wont do the trick.
I don't really know if it's just really hard til level 12 or if it was just really hard till level 12 for me because it wasn't until level 12 that I finally got rid of my Viper school silver sword.