I had this issue before and my main concern was the GWENT cards.
You end up getting the cards somewhere else; I can't remember where but if its because you have the goal of finishing the GWENT quest then don't worry, you'll get all the cards even without this guy.
*EDIT*
As far as I know this...