The downside would of course be that you might not get access to your leader card at all during a game which would kind of defeat the point, right? Instead of leader cards I would much rather have them just balance all the leader skills so that every leader is worth playing instead of just a select few.
The two biggest things that bother me about Gwent are the "battlefield" table top, and the 3d leader models. It makes the whole thing feel muddled like a mixed metaphor. At least give me the option to make my table look like a table and see my leader as a card... It's a card game.
Not a bad idea, adding a togglable option for leader cards that only on one's own side (by which I mean that the game gets shown with both using leader cards for oneself and the opponent still seeing one's animated leader if they do not want leader cards) leader cards are used would be the perfect compromise.
There is no need for the images both players see being the same.