People who have played the games first probably subconsciously project their experience with "Triss, the romantic love interest" onto her book counterpart, ascribing more importance to her role and generally attaching more value to the scenes she's in. Which is understandable, even if it slightly distorts the actual text.
Which I do confess to doing, being in the situation you prescribe.
Love is irrational. There is that certain chemistry between them in the games. Triss is smart in a cheeky way, and conversationally playful; what she wants is obvious, and her companionship and otherwise appeal is certain. The real problem is her lack of depth in character; we don't know her past very well, and certainly can't get nearly as well a grasp on her emotionally.
Then we have Yenn and Geralt. Everything in her life is by her own design. She was born deformed and probably unattractive, being made fun of and spat out like cruel, sadistic humanity loves to do. Her gift with magic was the saving and self-empowering grace that made her who she is. She's extremely intelligent, knowledgeable, and insightful...and also cunning, witty, and charming in a frigid sort of way that is designed to scare people off.
Geralt has his own share of problems. Frankly, he's obtuse, inappropriate, poor-mannered, uncultured, slovenly, and full of excuses...the 'I'm a Witcher' attitude is his answer to any issue he's uncomfortable with or unable to resolve on his own. The only times he decides he has important emotions are when Yenn leaves him.
Perhaps their irregularities are the key to their magnetic attraction. Why not any of the other hundred(s?) of her sex interests? He's unconditional in his regard of her; it's not reverence or fear. She admires his noble, self-sacrificing character. He's a witcher for god's sake, could be rich if he actually wanted it, instead he goes about seeing the world for what is is, not even trying to change it...and sees her and loves her for who she is, beneath the trappings of her life. It's reasonable to assume Yenn doesn't badger other partner's constantly like she does Geralt. 'Geralt take a bath, Geralt dress properly, Geralt loose the pretentious ponytail for once' and so on. It's hard to picture her tolerating the same level of garbage from anyone else. Even Istredd falls short, he's her intellectual equal, undoubtedly an interesting conversationalist, and a friend...but friends and lovers are different things. Geralt and her make each other better people, without forcing each other to change.
In the end, does it even matter if it is logical? For us readers, we'd like good reasons for the plot, but these books are written with ink not from the mind, but from the heart. Love is like vertigo, an adrenaline rush. A pure emotional wave lapping on the shore of logic, eroding reason and denial. The chord embedded at the base of the soul, plucked mercilessly for the creation of feelings verbally unknown and never fully understood. One of the, if not the only time, people feel fully alive. I won't pretend to be able to coherently, concisely explain why Geralt and Yennefer feel that way to me. It's a feeling I don't think I can shirk...and I don't want to, ever.