Basically after you did everything that Yennefer asked you to do. Quests in Skellige are more difficult than a lot of quests in Velen and Novigrad, so it's probably better not to get there too soon. I think the journal suggests level 16 for Skellige.