Are you facing an enemy sovereign with the intimidation trait?
No, i have the same problem. I fought against one wolf with my sovereign, her daughter some, champions and some elementals.
They got panic after the second round. My sovereign is cruel but that shouldn't have such a high influence. And does it take influence on the moral of my sovereign too? Thats not logical cause he knows he's cruel 
Oh and though i get away from topic does the moral have influence on my sovereign when casting spells? I always miss my spells...really...ALWAYS 
And please excuse my English 