While ideally it would be nice to increase game difficulty with a more competent AI, without altering game mechanics and keeping all options open to the player, that's unfortunately wishful thinking. If we want to play more challenging single player games, that means higher levels and tradeoffs to be made. To avoid excessive penality to player experience, these need to be based on an analysis of both game mechanics and AI weaknesses, not on what "should feel right".
I have been playing the game since 1.20 came out at both ridiculous (mostly wins) & insane (no wins) and simply sharing my analysis:
- the game is too "unstable" / "divergent", ie too favorable to the top dog and too punishing for the underdog, ie gives only benefits for being big (tech, production, resources, military, diplomacy...) without any drawbacks such as those found in civ (unhappiness, troop morale, inflation, diplomacy / reputation / AI wariness, trade / tech agreements...)
- AI being so weak at initial expansion, on ridiculous the player can attain top dog position fairly quickly and set the game's divergent dynamics going sufficiently in his/her favor that it overwhelms ridiculous AI "long term" perks (HP bonus, economics, ...)
- however, on insane AI gets such large "long term" perks that it can simply sit on its arse with a single settlement and win the game no matter what the player does, even if he/she has the rest of the map with a dozen settlements
- on higher levels, AI starts with bonus settler(s): it's more fun to have them start out strong and struggle to catch up rather than perk them up so much that they runaway no matter what you do (ie on insane)
is a suggestions for better balance. Again I am not advocating removing AI "long term" economic bonuses, simply rebalancing starting bonuses against long term ones to make the game more enjoyable on insane (and some of the other suggestions might make ridiculous more challenging, too)