Hmm got another crash after a couple minutes of playing the DX12 version. Got a crash message saying something about memory, before that VRAM hovers around 8000MB in Novigrad which is normal, then went up and up to about 12000 before the crash. (On RX 7900XTX, Driver Version: "22.40.00.62-230106a-387257C-AMD-Software-Adrenalin-Edition") Hope this helps.. I did send the crash report on the "Damn that's ugly" popup
Edit:
Support recommended me to do a clean reinstall of the game:
Back up save files, completely remove game, turn off save file sync, delete The Witcher 3 folder with settings and gamesaves on PC, check if the game folder in the install directory still exists and if yes delete it, install game (don't add mods yet), test if it works, move saves back into the gamesaves folder, re-enable save sync.
This actually helped the performance on DX12 a lot, it feels like there are fewer stutters and I get more FPS!

No idea what happened to my install that this made a difference, I had just upgraded my way every step from Ver. 4, verified game files a couple times as well...
Is anyone else seeing that VRAM usage keeps accumulating on DX12? It keeps gradually growing and only fast travel to another region or going to Main Menu and back into game 'resets' it for me.