Except you can hardly remove anything else. That is the main problem. The fear that the opponent might be running this card stops you from resetting and locking his STR 21 Nekkers (it did so last night for me), his Vran warriors, anything really. Not to mention that I am now officially as tech'd against this card as I can be and still lose to it (lost to it twice today, once when I didn't have both removals/locks in hand, another time when he Adr. Rushed Imlerith and buffed him to 16 STR when all the removal I had was Seltkirk). And if he proves to not have it, and you kept your removal out of fear, well, too late now, the Nekkers are in the stomach of the immune Arachas Queen and have all been brought to the board. This is completely binary guessing. If you decide to keep them and he doesn't have Imlerith Sabbath, then you've lost because you didn't use them when it was a perfect opportunity to do so. If you use them but then I:S appears, you will also lose if your opponent pulls Mandrake (or if you use something with weak bodies, even if he doesn't pull mandrake he can cost you the game) Only if you guess correctly whether your opponent has this card can you win. That's hardly strategic.
Of course having to hold onto removal is not the game's problem. Having to hold onto it even when you could use it to great effect simply out of fear is however bad. Like you said and I quote, "there are tons of decks that require you to hold onto removal because [you] KNOW they very likely have something horrible in the offing if [you] don't". See the problem? With Imlerith you have to hold onto it for the mere possibility. You don't know that they have something horrible. It might even prove to be that they don't have anything at all and you held onto your removal when you could have used it to great effect for nothing but fear of a possibility.
And to answer your last question, well, it's that the context is very narrow. Imlerith out of context is not bad at all. However to answer, I must bring the context of use in the discussion because there is no single card in the game that can get out of hand without context. Greatswords need Longships and a long round. Enforcers need spies, need to be played early, and a long round. Mangonels need revealers, they need to be played early too. Villentretenmerth needs a good line up of targets and 3 turns, etc etc.
Imlerith however, all he needs is Mandrake. You can drop him on the penultimate turn and Mandrake him and he'll still be out of hand, especially if you have the last card over your opponent. He gets out of hand far faster than ANYTHING else. Within 2 turns (one to drop him, one to Mandrake him) he is out of control unless you can scorch him (and the opponent doesn't have Renew of course). No other card gets out of control, even with the context of use, that fast.