People here often mix up difficult with annoying.
e.g.: Tougher gameplay with more harshly punished mistakes? --> difficult
e.g.: Revisit every spot you visited before again for better gear? --> annoying
Ultimately, it's up to the developers on how they fill this in, but me personally, i'd let the player keep all things that were grinded: skills, points, all items (except quest), gwents, all discovered locations (though not the items within them), alchemical formulae, bestiary and character description, armor/sword diagrams
To fill in difficulty I'd: Change the way enemies level (ie, never let them sink below your level, except the ones that are higher than you, those remain higher), change enemy patterns (eg 2 ekkimara's instead of one, or one stronger fiend supported by more smaller enemies instead of just the fiend) or place them differently like more archers widespread...
Of course, this is all easy talk. Because during development of this awesome, awesome game, there is never enough time or money nor pleasing everyone. SO in the end, it's the developers which choose how they go about doing what they actually CAN do. There's three things to a service: Cheap, Fast and Good, and you can ALWAYS only pick two.