I am on my 3rd Playthrough on Xbox One and on every playthrough i had the same issue. I came to the place when i was to weak and later in the game, there were everytime only 4 bandits on horses and none of them had a key.
Otherwise i have to say, if there is nothing missable from one of those persons, i don`t care if its fixed or not!