The Toad was the worst of them all, for me, because I couldn't figure out when and where it would jump - which it did an annoying lot, especially for a melee focused build - and overall I didn't feel very confident with its visual cues, hard to anticipate what it would do. Plus Geralt's contradictory conclusion that it doesn't mind fire, while actually it's one of your best tools to incinerate the gas. I resorted to cheesy stuff like freezing it time and again, without feeling I had actually mastered the encounter.
I found that he only jumps if he's out of range or if you're trying to chain attacks on him. When you're out of range, he'll always jump towards you, but when you attack, he always stomps, which knocks you into range of his tongue attack. The breakthrough for me was realizing that his tongue attack is actually only mid-range (it looks like it can hit a lot further out then it really can), so you can lay an Yrden and bait him in from far away. Even a basic one should do. When he's locked down, you can get in 1-2 good hits, dodge the stomp, then rinse and repeat.
I prefer Golden Oriole to burning poison clouds, since I tend to blow myself up whenever I try that. Plus, you might as well get the cheese healing (I feel like he can't actually kill you if you have GO on). I suppose bomb-focused builds could probably rock this boss pretty hard as well because of all the poison he uses. Also, I didn't know the toad was a Cursed at the time, but I guess you could apply your own poison DoT effects on him this way.
The main problem with Caretaker is the fact that he stuns and knockbacks which allows heal to hit you and heal.
Use the decoction which gives you near immunity to stun and other effects. He knocks and you instead of 4s, it takes 0,2s for you to get back into the fight. Keep quen up and thats it.
Olgierd ghosts are much harder because they do massive damage. Few things. Play it safe. Constantly dodge, roll, keep quen up. On the last two ghosts be even more careful because they can oneshot you without quen.
I'd also recommend the auto-Quen decoction vs. both these guys. Both bosses are all about heavy stun/stagger effects and big burst damage.
From what I've read, there isn't really a good way to go toe-to-toe with the Caretaker, so the best thing to do is to wait until he stuns himself with his AoE attack. He also has trouble dealing with obstacles, so it's good to keep something between you and him until he does said AoE.
The Olgierd ghosts can be dealt with handily if you've got good ranged attacks. I burned him down with Igni, but a cluster bomb build could probably wreck him quick. Come to think of it, cluster bombs would probably wreck all the HoS bosses pretty easily.