Yeah, that's the thing that may get you. You can reject quests and go back to say "yes" later, but be sure that you finish what you're doing once you start it. Once you advance the main story line past certain points, certain quests become unavailable or auto-fail.