Without more information it's hard to diagnose the issue but I can take a guess at what might be causing it. If your hitting escape as you are moving the mouse around that will cause the appearance of the escape key not working. Another alternative to closing the inventory or any other information screen is simply hitting the same key used to open it instead of escape.
