I disliked the beta method a great deal. Sure, it stuck with the theme better, but it REALLY hampered the way that I like to play the game, and my enjoyment (is that the right word for Beta?) of the game was impacted because of it.
I don't think any of us advocating this are trying to 'punish' big kingdoms or take away that strategy, but rather allow smaller kingdoms to viable -- we're trying to add a choice, not substituting one choice for another.
If small kingdoms are to be viable then there must be a way to balance size, and essence is a good candidate -- it's powerful, it's lore, and large kingdoms can substitute the quantity of troops that having many cities allows for a corresponding decrease in their magic ability (while still retaining significant magical abilities, just as the small would still retain quality troops, just not as powerful/many).
I absolutely agree that the current AI's absolute maniac compulsion to fill up every last available square of land without regard to resources, strategy, or apparent reason/objective drives me absolutely batty...
With all the AI pioneers running around I feel like I'm stomping cockroaches. Where you see 1 you know there's hundreds skulking about.
There's an enemy pioneer! *stomp*
Another pioneer *stomp*
And another *stomp*
Same with enemy cities -- *stomp* *stomp* *stomp*
It's more tedious than fun (and the game is otherwise rapidly getting to be much much fun!), and it makes pursuing the other aspects of the game less viable, as ignoring the cockroaches has dire consequences later on.
...but as a player who doesn't build a city without some form of specific reason (choke point or resource), hampering my play-style's ability to expand would be just as frustrating.
The decision process you use for cities is a good one. The AI should have a similar one, and not just rush to place as it apparently now does.
There's really 2 ideas here -- small being viable vs. large, and the AI's mad expansion. Addressing the former then having the AI take that into account should 'fix' the latter.