That's the thing, she can get sex wherever she wants, but Geralt is the only person who will ever love her. She is, however, so used to and content with being a tortured soul, that she's willing to live with on-and-off-again affection from him and sees no reason to change (much). Sure others might be infatuated with her, or love the idea of her as they see her, but only Geralt ever saw through her and accepted her for who she was...personality defects and all.
In the books, where Geralt is the product of one man's mind and imagination, that is correct. But in the games, each individual player brings their own thoughts and experiences to - much as I dislike the term - "their" Geralt. I used to do a ton of theater. Unless the writer was actually sitting there with you, it was up to you - as the actor - to "find the truth" in the character for their portrayal. Two actors can play the exact same character and draw out completely different emotions, depending on what they bring.
In game, default Geralt has a history with Yennefer. He has memories of the good times, he knows her personality, he knows her. We as players do not have that. I've played the games, I've read the books, I even watched the TV series. I can still see that Yen treats Geralt horribly. Even the character comments on it. So maybe... maybe... Yen is not a horrible person; maybe that's the way I see it because it's the eye of the beholder. But here's another example...
Let's say that I have some friends. People I've known for years. We act like absolute dicks to each other. But we know each other, we know it's in good fun, we know they're jokes, we know that no matter what we say that it's all good. Now through some random act, you are me for a day. You look like me, talk like me, but mentally... you're still you. If a bunch of guys - that you know you have history with - come up and treat you like crap from the get go... how do you feel? You're lacking the context of why we act that way. The same thing goes for the game. I knew going in that Yen was a handful, I knew it was "Geralt and Yen 5 ever". But I have no memories of her being caring, being loving. I have no memories of her being anything but the title of a famous Jet song.