Not only is there no reason to decline a quest, but if you decline it, the quest vanishes for good.

What I'd really like to see are quests that remain behind if you decline them and actually get worse.  Don't want to deal with buchermen who captured that logger?  Perhaps later he could use what brains he has left to gather another butcherman or two and become a haunter.

I have chosen to end a quest that was beyond my capability (at the time) because the destination spot had appeared directly on top of the most fertile tile in the area and I needed it to plant a city.

There is like one strong quest that spawns a monster on the world map as you accept a quest.

Until you realize it's a dirty big dragon.