The downside would of course be that you might not get access to your leader card at all during a game which would kind of defeat the point, right? Instead of leader cards I would much rather have them just balance all the leader skills so that every leader is worth playing instead of just a select few.