I had two outposts separated by a third AI outpost in the middle - so one outpost of mine, one of theirs and then one of mine again.  I build a road through 3 squares of their outpost.  Above their outpost was impassible terrain (a mountain IRRC).   So any troops going East to West  had to dip into neutral territory for half of one movement and then continue on their way.  The units would not end up in neutral territory, merely pass through it.

The pathing would not do this.  It would continually re-route the units the LOOOONG way around the mountains adding several turns to a simple maneuver.  I couldn't even target the 1st square of my distant outpost from there without the little feet of pathing refusing to accept the shortest route. 

I damn near declared war on the neutral AI right there and then because of that sillyness every single time I wanted to move a troop across my empire.

Making non-hostile units stackable is a big deal.  If an attack happens who does it happen against?  How would I know if there was a unit I was at war with in a tile with a unit I'm not at war with?  Some would suggest not allowing them to stop in the tile but allowing them to move through the tile, which is also a major change (at this stage of the project).

So we are unlikely to allow stacking units because it will open up a host of new issues that are worse than the problem of having tiles blocked (ie: the cure is worse than the disease).

I am having a dev checkout switching it so that you can move onto or through improvements of neutral players (without destroying them).  There may be some technical issue that was added for, but if we can do it without causing more issues we will get it switched.

I think the questions you pose above are answerable. I'm not sure I entirely understand the difficulty, because under the current mechanics, you cannot initiate an attack against a unit unless you are hostile to it. The default cursor option is "talk to", not "attack". In order to attack a non-hostile unit, you must first declare war. One could have the option of "talking to" or "moving to", which shouldn't be that bad.

But if there is a stack of units, and some are units towards which I am not hostile, and some are units towards which I am hostile, then moving to or through that tile would no longer be allowable. Clicking on the stack could simply give me one of two options, since "move to" would no longer be valid: "talk to" or "attack". If the latter, then (obviously) battle would be initiated only between armies against which I am hostile. This is currently the case as well, when two or more hostile stacks occupy one tile.

In other words, IF a tile contains ONLY entities which are flagged non-hostile, THEN commands "move to" (and "talk to" if a unit and not a structre) are valid. IF a tile contains more than one army, THEN a number appears above to denote this; IF a tile contains more than one team, THEN a context-sensitive selection is offered for the command "talk to". IF a tile contains any entity flagged hostile, THEN command "move to" is invalid, but "talk to" is valid and offers the same context-sensitive selection as above. IF a tile contains any entity flagged hostile, THEN command "attack" is valid, and the same mechanic is used to select the target for battle as is used currently, when two or more hostile armies occupy the same tile.

I find the disease truly awful, and while I understand that the cure is more resource-intensive than I had originally thought, working out a way (via drop-down menu or context-sensitive mouse-over) to select a valid command (move to, talk to, attack) at a valid destination should be doable.

No?

One last thing: Currently, if you set mouse-over to show the path a unit will take, and click on the location, you will often be dissapointed to see your unit take off in another direction, because the computer will reckon the path differently (based on a non-hostile entity blocking the way) -- and you cannot stop that unit from moving in the wrong direction and wasting a full turn until the unit comes to a stop