Vilgefortz can also pull a high strength card, like Old Speartip. Or even worse a card with Order that the opponent can use directly. If you fish for Syanna is an insta-loss for the Vilgefortz player. From the Imperial Manticore to Windhalm an Attre, to really anything with a decent-body (Barbegazi, Ice Giant), there are a lot of cards you don't want to see. Ruhein can ruin your day. Really anything with deathwish or that ticks over time like Sly Seductress is suboptimal for the Vilgefortz player.
Now compare Vilgefortz with Yennefer's Invocation. What is the downside of Yennefer?
You take the card directly, which you might even play the same round with the leader ability or next turn with Yoachim or Roderick. That's the reason why you see Yennefer's Invocation, while Vilgefortz it's not so common.
Better a 4 pt poison card than Vilgefortz on the melee row. If you have Vilgefortz in your deck, is mostly for its ranged ability...
I would argue, that Vilgefortz is the sort of cards that Gwent needs. It's powerful, but it has a downside and the player has to think when it is the right time to use it. It can spell doom for you, if you pull something worse than the threat you just removed. It's a card you can use offensively or defensively, which is also good design.
Except this kind of unfortunate pull doesn't happen all that often (most cards are about deploy), but even when it happens, it disrupts enemy's game plan due to releasing a card before it's supposed to go. Imagine pulling out Syanna before the proper setup is ready. Sure, the ability can be used immediately, but probably shouldn't be, chances are, it's too early for that (Regis/Aglais/whatever else that needs a lot of setup) -, or, perhaps, too late for that, because the R1 is basically over, and now you either have to use your R3 combo in R1, or deal with the consequences of not having the full combo for the finale. Even Speartip and Imperial Manticore need proper use to be effective in spite of their "simplicity", but Vilgefortz denies that, so "backfiring" can't really be an argument. Having them released too early against a faction full of 1-tap-kills is a generally bad idea, innit? If you pulled Ruehin, you can easily YI him before your opponent gets the chance to raise any value from him, so he isn't exactly a bad catch either.
So in the end, Vilge:
1) Kills any units unconditionally
2) Most likely ruins one more
2.5) If you're unfortunate enough, it also ruins your entire strat while he's at article 2. Enjoy your Regis, man.
3) All for 9 provisions
The reason he sees less play than classic poison stuff? He's "expensive". Well, I mean, too expensive for spoiled double ball players, since they have so many easy removal options. Another reason YI sees more play is sadistic pleasure they derive from using your key cards against you. And lastly - maybe - they are afraid of pulling something bad, even though they really shouldn't be with a toolkit like that.