You do not even need Quen for Wyverns, hit the Wyvern 2 times, dodge and repeat.
I am playing on the 3. difficulty, and the difficulty won´t do a thing to Wyvern, they are just to easy because of their behaviour. Griffins or Gorgos are way harder and my most hated enemy are werewolves.