If you're having this problem and are dumb like myself, this might help you out.
There are TWO crystals to the portal you have to use Aard on in order for the portal to work. The one on the left is the crystal you found on the ledge and the one on the right was already there.
