I started a new playthrough yesterday and I cleared all the abandonded sites in White Orchard. I did so in 1.2 and I did so now in 1.6 .... I really wonder, why it is bugged in your case.
Some POIs just bug out randomly. I had numerous abandoned sites bug out on me in one playthrough while on others they were fine. Can't really pinpoint what causes it though.