The transfer rate between GPU and SSD is no faster than for other M.2 SSDs. Still, by using it as some sort of additional cache of RAM, latency for accessing this storage pool is a lot lower than going the normal route to your system drives. I think you can compare the situation to a hybrid HDD with some gigabytes of flash memory for reduced access times.
In CUDA applications (and probably other GPGPU API's) the latency of memory operations is hidden by properly scheduling reads and writes to and from device memory. Noone (maybe students?) writes on device memory, reads back and then loads files from slow storage such as hard drives. Worst case scenario, if data doesn't fit in RAM, it should be preloaded while the device is busy. I can't imagine an SSD being preferred over RAM, specially now that it is cheap and proper workstations and HPC nodes have hundreds of GB's. SSD's fall a step above mechanical hard drives in memory hierarchy but still below RAM. So if slow, permanent storage is still necessary why not just load every rendering node with a few massive SSD's, a few massive HDD's and the fastest Infiniband connection available to a humongous storage server? We're talking about multi million dollar investments here, not garage operations. Unless that's AMD's target audience since Nvidia is all over the place in industry, research labs and universities.
There are other, much more serious issues affecting the progress of GPGPU technologies for anything other than data-parallel applications with algebraic solutions. For instance, transparent and more direct memory operations and more control over different segments of the SP array..