Ok, this is a very annoying mechanic for several reasons. Not only we have the throwing knife perk (Dagger Dealer) but also there's another perk associated to thrown knives (Cutthroat). If we put into the equation perks like venomous fang and silent finisher, we can see that knife throwing is something planned from start.
At this point there are 2 solutions
1- Making knives retrievable: of course the dmg done is quite high, but in the meantime, while you are not holding a knife you are unable to attack unless you use another weapon (which makes knife throwing pointless to start with) or slotting other 3 knives which gives you the chance of 3 projectiles. Since it's an "all in" mechanic, missing a shot implies high risk even in normal difficult. I think that solution is the most balanced one (if you want a pure knife thrower)
2- Stacking knives: It seems legit, but the amount of knives in the game is quite low, their value is expensive for a one shot mechanic, and the dmg, being high, would make this pretty OP. To make it more balanced it is necessary to increase the amount of throwable knives in the game, reduce their cost for making them/buying them, and reducing also their dmg. In turn the mechanic will give nothing extra since knives will be behaving exactly as ammo while you are the weapon. There's a reason why in Diablo 2 few people dared to try throwing builds instead of good ol' bow and arrow. I hope you don't make the same mistake.