Considering that the silver sword is for post conjunction monsters and the wild hunt would theoretically fall into that category, since they can only cross dimensions via magic/portals, I dont think its strange at all, despite them not being specters, unlike in TW1. Besides - the fact he only had his silver sword left didnt keep Geralt from killing Jacques de Aldersberg in TW1. The steel sword is merely more convenient for opponent who dont require a silver sword to be killed. Silver will still get the job done though. A sword is a sword after all.