There is that, but also... One of the biggest things humans have used their imagination for throughout history, is how to make other people suffer, whether they are adversaries or rejects or what ever. And it echoes today still. We don't have coliseums anymore where tigers maul the gladiators, but violence and suffering interests people still, sometimes to unhealthy lengths but nonetheless. People tend to prefer story settings that are dark and where people are miserable and get hurt (physically and/or mentally); people tend to relate better to the bad guys in stories, how evil and cruel and insidious they are - rather than to the altruism of the hero. I.e. every Bond film is defined by how "good" their villain is.
Perhaps that is so that there'd be a point of reference so we could actually feel "good" when there's reason to, that joy doesn't come for granted. I dunno, I'm not a psychologist, but there is a logic to be found there.