It doesn't "degrade over time" - it escalates.
Meaning, you will get closer and closer to the ceiling on max settings - depending on where you are in the game and what you're doing.
Some areas and activities seem to take up more memory - and, at some point, you will reach the ceiling for the...