Both your Min & Max are lower than my results so something is definitely wrong, My results are something like this with Ultra preset, SMAA and Ubisoft fur at 1080p.
Min: 42 - 45
Max: 65 - 70
I haven't gone below 40 except when I use Nvidia's fur (Hairworks) or MSAA. The biggest hit I took was at Banapur (first village) and even there it was like 34 or so.
.
Yeah, don't know what's the problem. I have 14.11.2 beta drivers installed but it seems to make no difference