I didn't do anything with Gwent on my first playthrough and my word did I miss out now i've started my second on Death March difficulty and Gwent is just a great addition and the extra XP is very much worth it in the lower levels, gives you that boost here and there.
Would definitely recommend...