I would just recommend you go for whatever you think looks the best, the game isn't really hard enough to warrant min-maxing.
I usually go for Kaer Morhen armor as my default, but take off the gloves and put on a shirt when hanging around in Novigrad on hot days, basic feline armor also lends itself well to sunny weather. In Skellige I sometimes go for the Bear armor, because it looks so nice and warm.