He was one of Red Riders for 2 years... he knew what they are about...
No he doesn't. He clearly states he doesn't remember most, if anything, during his time as a rider of the wild hunt. Avalach is the only one who actually tells Geralt that he fought Eredin's projection at the lighthouse in the world destroyed by the White Frost, and that Ciri fought him aswell to save Geralt from the Hunt. In fact, if you don't know much about the books or the red riders, Avalach's explanation makes you think Ciri is being hunted by Eredin because she took Geralt from his clutches, which ends up being a contradiction to the whole "I wanna kill Ciri before her elder blood offspring destroy my world" thing.
The Wild Hunt was haunting Geralt in the 1st game meaning they always wanted the Hunt to be a highly significant plot element for the trilogy, and when they do make the 3rd game, which happens to be titled Wild Hunt, the Hunt plays more of a secondary role for a large part of the game. They only show up chasing Yennefer and Geralt after White Orchard, and then disappear until you find Ciri in the Isle of Mists. So, suddenly after White Orchard the Hunt loses all kind of interest in Yen and Geralt? What's the point of that cutscene with them being chased by the Hunt? It would make sense they wouldn't do anything while you're running around inside Novigrad for example, seeing as that would draw unwanted attention on them, but not haunting Geralt in Velen, even if only on spectral form doesn't make much sense imo.
In one hand we have the premise that Eredin doesn't really see Geralt as a threat, which doesn't make much sense after what was seen during W1, but i could understand that. In the other hand though you have the fact that the Hunt actually tried to catch or kill Geralt and Yen while they're travelling to Vizima, to meet the Emperor and learn that Ciri is back in the witcher world, but once they do get there, the Hunt doesn't try to stop them anymore. To me it seems inconsistent... At least having Geralt try to learn more about his time as a Red Rider, when his amnesia was such a big deal during W1 and W2, would seem natural. After all humans are curious by nature, and even if Geralt is a mutant, he's still human... Eredin also clearly showed an interest in him and his actions during W1, so far as saying that in the end his choices only ended up creating more pain and suffering for those involved. But then suddenly Eredin doesn't care about him, his actions or even see him as a threat even though during the flashbacks on W2 Geralt is seen killing many riders once they take Yen, and later kills lots more at the Hanged Man's Tree with Letho and the other 2 Viper school witchers. Eredin even goes as far as calling him his greatest champion in W1 due to the chaos and death that followed him, and yet he didn't think Geralt would try to foil his every plan to catch Ciri? That he wouldn't do anything to defend her? A villain that is supposed to be a conqueror and a powerful military leader that does not try to stop the single BIGGEST threat to his plan seems rather idiotic. So in conclusion, Eredin is either supposed to have the intelligence of a chicken or was outright badly written. The way the game is right now regarding the Hunt, Imlerith is much more a villain than Eredin could ever be, seeing as him killing Vesemir has a much greater impact in the story than Eredin ever does with his bad one liners. Hell, Imlerith killing Vesemir makes Ciri lose control and nearly destroy Kaer Morhen along with everyone she deeply cares about.
Eredin is just too passive to actually have any relevance to the story as it is.
Regarding Letho and his disappearance for a big part of W2, it's different. It actually made sense storywise. Letho was for all purposes an assassin of kings. He wouldn't be a very good assassin, something that would assume a certain degree of stealth and deceit, if he was always showing up around every corner. With the Hunt should be the exact opposite though, they should be much more visible since they're considered an omen of war/natural disaster by most while some don't even believe they're real, both things that any good military leader would use to his advantage, even more so considering the witcher world is in chaos and at war.