Turns out it was a bug that can be fixed via console.
here's how.
type
DBGConsoleOn=true
under [GENERAL] in the C:\Users\YOUR ACCOUNT HERE\Documents\The Witcher 3\user.settings file.
After that, open the console in game via the tilde key and type enablefasttravelpoints=1 or...