Non-aggression pacts should NOT allow another kingdom to waltz through your territory and set up a new city in between your other cities. Canada has a pretty close relationship with the US but we don't want you waltzing up north (where few people live because it's so cold heh) and setting up shop, that's just not the way it works.
Non-aggression pacts should have big consequences if you break them, but shouldn't allow other kingdoms troops to cross your cultural boundary. Also, countries I've never formed any agreements with should not be allowed to have caravans waltz through my territory.
It wouldn't be a big deal, but I like big open areas in my empire because I farm the crap out of them for gold and exp. Signing a non-aggression pact shouldn't mean I now have to kill a kingdom I had cordial relations with to get my environmental/wildlife sanctuaries back 