Is there any reason whatsoever to decline a quest?
If there is no immediate threat in accepting a quest, that is, you first have to move elsewhere to start it, what is the point in not accepting it? You can keep all those deadly and epic challenges until late in the game or not tackle them at all. Age of Wonders imposes a turn limit on quests with mildly bad consequences if you fail, and that seems quite reasonable to me.