Yesterday my Fort (with Fort) was attacked and the battle happened for some reasons outside the town.
After the battle the militia was together with my heroes outside of town and marked as 4 armies.
My heroes were at the same time also in the town.
I moved my army into town and that seemed to fix the issue.
But two rounds later I was attacked again and now only the militia unit with bow showed up for the battle (the other may have been killed in the last siege).