Open with notepad (or similar plain text editor) the following file <user>\Documents\The Witcher 3\user.settings and change "TextureMemoryBudget=" to a value in MB that you want to allow for texture caching, but do not set it to your maximum since there are other things than textures that need memory. I would probably not put it above 3072 MB, but that is just my opinion. I definitely would not put it above 3584 MB so that you have at least a theoretical 512 MB left for other things. There might be a hardcoded cap much lower than that, and ultra only sets it to 800 MB. There also might be assumptions in the engine that break down when that much memory is allocated, or it might work fine. I have compared load times between 2048 and 800, and it takes 5-10 seconds longer usually to load a save. I have not tried higher values, and I also have 4 GB. The game is currently using 2 GB of gpu ram while that value is set to 1024, so there is almost a GB of "other things" loaded up right now. Windows is also using 200MB or so of gpu ram. So really, 2048 is probably the highest you want to put TextureMemoryBudget, or you could set it to 2400 (just triple what it sets to at ultra). I am used to using powers of 2 is why I set it slightly different. The engine may be smart enough to not use more memory than you have, and to free up space when you run out, but I would not risk it since a lot of engines are not that smart and just crash if you set things too high ie set it the full 4096 and likely the game will crash at some point.
The curse of hand-made stuff. On the other hand, therefore you don't get copy-and-paste stuff everywhere and mediocre textures throughout the game.
I also hope for a high Texture Pack though. I can't enable that while recording the game but I sure as hell would like to enable that when playing the game on my own at a later time.