Tie between Operator and the Beast. Operator is just plain hard, didn't get him the first time on easy without cheating.
I think I got him once on hard in the meantime, though.
The Beast is so random... definitely the most feared boss for me, I always dreaded the end of Chapter I. I cursed a lot, I died a lot, even with perfect character build and perfect preparation (preparation at 22h at the aard circle of elements, then potions and buffs in place at midnight). But imagine how anticlimactic it is, with all that preparation and dread, then you go up to it, stun it with the first aard, finisher and The End. That also happended to me once and I was dumbstuck and almost mad, lol. I think I even might have been so surprised that I let myself get killed by the mob afterwards and did it again anyway.
The dragon fire in the prologue is also random, live or die by luck alone. Meh.
Kayran is not a proper boss, I hate the thing, that fight somehow doesn't fit the game at all.
Letho is hard, yes. He has the same skills you have and more, because he doesn't have pauses between his actions. But it's a large area (we're talking 2nd fight, right?). You can evade a lot, wait for him to drop his quen, always be two steps ahead of him (because of your pauses), use bombs and magic (am I the only one, or does his igni have three times the range than ours?). Patience is the way here, not a luxury you have with the Operator.