Pretty much agreed. It really is a problem because of the eXpand point in the 4X. You always expand without any limiting factors, growing more and more cities, which affects the terrain(not to mention micro-management and overall atmosphere).
What I would like is that cities were much fewer in number(a bit like in the HOMM series, where you can't build more of them) and the wilderness had more of an impact overall. Perhaps instead of full-blown cities you would have small villages that can never grow into cities(thus limiting out of control expansion), and if they were attacked by roaming monsters if too deep in "barbaric lands" it'd also limit their number, effectively making more wilderness.
Or just have a lot of terrain that is poorly suited for a city, perhaps even so much that making a city there would be foolish, because it costs resources instead of gaining them. Then you'd have to actually think if you want to make a city somewhere, perhaps for defensive purposes or to gather some rare resource(if they are included).