Playing games on difficulty settings ranging from normal to challenging to hard, I have never had the AI declare war on me in something like 5 or 6 games (except, obviously when I walk into their cities with a cheesy rock giant). I play aggressively though. Usually the AI are busy defending against me and my army of cheap, overpowered summons.
If you're not going to play militarily aggressively in games like this, then, as in real life, you *must* have a well developed military deterrent, and you must aggressively pursue *diplomatic* avenues to stave off invasion. Develop treaties, trade routes, exchanges. One way or another, even if you are not invading, you cannot ignore the AI, leave your military undeveloped, and hope to keep their greedy hands off your land.