It's not a bug, the "clean shave" beard does grow back, it just takes time. I transitioned from Velen to White Orchard and it grew back.
Edit: If you rested for a day or two and needed to shave again, that would be tedious and not fun if you wanted to be clean shaven, I suspect the long wait...