It depends on the game and on what is it that is "difficult". "Difficulty" usually means the number of enemies you fight and how many blows it takes to kill them. In a game like Witcher, fighting gets in the way of my enjoyment of the story, the setting, the attention to detail in the environment... Now, there are some games which bring some other elements to the story not seeing at lower difficulty levels. Depending on what those elements are, I may choose to "crank it up". My enjoyment of a game like Witcher is following the story and seeing how the decisions I make affect the story: first playthrough I decide as I would normally do, then on subsequent playthroughs, I make decisions one way, then the other way... frankly, all the fighting sometimes gets in the way of how I enjoy the game.