I suspect it is the same problem that with stealth: the monsters are not attacking. They are just walking carelessly and step on your caravan! It seems that, even if the monster does not see you, if he walks randomly in the position of your unit it still starts a fight. It sucks, but I am not sure it has an easy solution 