This issue appears with all quest were you have to fight enemies (eg. Dragon questline, The forge questline). If you lose or even flee you are still awarded the quest reward, and the monsters are simply placed on the map.

You should either

- get the possibility to return later fighting said group of monster (this could be bad because a different faction could kill them)

- get the possibility to start the fight again somewhat later (monsters should then not appear on the map)

- simply fail. The quest is removed from the questbook (this could also be bad cause it would make the master quest fail, e.g. at the second step).

On a related note when doing quests with were stong monsters have to be confrontet, the quest dialogue only gives the opportunity to fight. At this point it would be preferable if you could also choose to retreat and return later with stronger force, and be in fact able to do so. This is important because for some quests you are not warned upfront, how much resistance is to be expected [many quests offer no resistance (combatwise) so one get easily a bit lazy].