So I remember that in TW2 it was said that Geralt rode with the hunt. and now I did the Battle of Kaer Morhen and he mentions it again, but I don't understand how it is possible? Any background information from the books on this?
Nope, Geralt never encountered the Wild Hunt in the books. Geralt being taken by the Wild Hunt is CDPR's invention, which of course made it so they could do a game about him.
Geralt was taken by the Wild hunt after the end of the books and before the beginning of The Witcher 1, this is explained by the TW2 flashbacks: https://www.youtube.com/watch?v=imh-aZmWoAQ