Every time you talk to a new merchant, play gwent until you win.
That way, when you run into a merchant that has ANY grey text, you know you played them already.
And yes you will get crafting materials from a win where you already won a card.
Oddly, if you have won all the winnable random cards, and you play someone for the first time, you get nothing but the crowns you staked.
Obviously you don't get a card, but you wont get materials either.
This will tell you when you have collected all random cards.
If you play a dude and he gives you nothing, and you go talk to another brand new merchant and win nothing, you can be certain you have won all the cards you can from playing random people.