I'd love there to be a default diplomacy status where the neutral factions won't wonder into your territory, and there be a diplomacy option for you to tell another faction to gtfo out of your turf. At a cost of relations of course.
I just find it annoying that sometimes a potential enemy has a few stacks of scouts near my resources where they can simply pillage and ruin my economy.