1. I have only 1 army outside of my cities, nowhere near any of their cities.

2. I do have caravans out that travel near their cities, not sure if this triggers it or not.

3. This is very annoying, since it's happening so frequently, if they hate me so much, declare war already!!

4. I'm the only empire faction in this game(that I know of), the other 3 I know are all kingdom, so I am not very well liked

This is happening in the same game I was having the disappering city. Later, the outpost that was near me from Kato faction dissapeared, and the Lealickan faction that keeps pestering me, built an outpost right next to my border.