You have to buy the maps from merchants around Velen/Novigrad (I haven't been to Skellige yet, so I don't know if there are maps there).
Once you read the maps, it will unlock the Scavenger Hunt quests (In parts).
---------- Updated at 11:36 AM ----------
Just to add merchants, the ones with...