**Fix for PC**: So a kind of work around found by Sanguiss. By installing the console mod (
http://www.nexusmods.com/witcher3/mods/28/?) and then typing xy(718,2079.36) you should teleport to inside the door to the manor. If you spawn in the air just open the console and press the up arrow then enter the command again. You should eventually spawn in the door and be able to pass through. Unfortunately I still have no idea how you might solve the problem on PS4