I went to a level 3 quest location (I believe it was called ancient ruins) with a weak character and it asked me if I was ready to fight. I responded No, since I didn't think that character would survive a fight, and woudl come back later. The quest hut disappeared, and I was a little bummed. But what is worse is that all L3 quest huts don't work at all now. I can run all the L1,2, and 4 quests, but the L3 ones just don't work. I walk on them and it is like the tile is empty.
I can e-mail a save game to show you if you would like.