I know its a bit gamey, but I really liked how in Civ V you kinda just weren't allowed to enter hostile territory without a declaration of war or a treaty of some sort.

I really wouldn't mind it so much if it was a quid-pro-quo type of arrangement, however if you go into one of those AI territories, they are quick to tell you to get out. Telling them to get out every turn is painful (as well as it I suspect it changes their outlook toward you - suggestion: maybe once you do, they won't try again for 20 turns).

I would rather have a Open Borders arrangement that you can negotiate with a turn maximum (say a 10 turns version and 99 turn version), but otherwise the AI stays out.

As it stands now, it seems like the developer's original idea was "The player has to defend their borders, so you can go wherever you want." But then they realized this was not very fun (and frequently frustrating) and reached an internal compromised that "Well, the player can make them leave if they want to, but we still really like the idea of having to defend your borders." The problem is that this just creates needless busy work and is not fun. Devs, you cannot have it both ways.

You should not be allowed to enter someone's borders without an agreement to do so.