While ideally it would be nice to increase game difficulty with a more competent AI, without altering game mechanics and keeping all options open to the player, that's unfortunately wishful thinking. If we want to play more challenging single player games, that means higher levels and tradeoffs to be made. To avoid excessive penality to player experience, these need to be based on an analysis of both game mechanics and AI weaknesses, not on what "should feel right".

I have been playing the game since 1.20 came out at both ridiculous (mostly wins) & insane (no wins) and simply sharing my analysis:

- the game is too "unstable" / "divergent", ie too favorable to the top dog and too punishing for the underdog, ie gives only benefits for being big (tech, production, resources, military, diplomacy...) without any drawbacks such as those found in civ (unhappiness, troop morale, inflation, diplomacy / reputation / AI wariness, trade / tech agreements...)

- AI being so weak at initial expansion, on ridiculous the player can attain top dog position fairly quickly and set the game's divergent dynamics going sufficiently in his/her favor that it overwhelms ridiculous AI "long term" perks (HP bonus, economics, ...)

- however, on insane AI gets such large "long term" perks that it can simply sit on its arse with a single settlement and win the game no matter what the player does, even if he/she has the rest of the map with a dozen settlements

Therefore:

Sure.

Engineering Faction ability (cost 1) : unlocks 3 unit design abilities, at Labor cost.
- Builders will allow the unit to build an outpost. Unit disappear once the outpost is built
- Settlers will allow the unit to settle a new village. Unit disappear once the village is built. The ability is unlocked with Civics Tech
- Road builders will allow the unit to .... well, build roads. The ability is unlocked with Trading Tech.
Note that the basic Pioneers (at Population/Gold cost) are not modified but still useful.
Fortifications can be built in all the Engineers cities, and they're always upgraded (more troops + unit bonuses). They become available as soon as a workshop is built in the city. Only the Fortress city type can build Fort / Castle buildings, which are also upgraded (more troops + unit bonuses).

So ... Engineers can train units able to settle a village, to build an outpost, or to build roads. These units can wear armor, weapons and magic items, and they can have mounts and other traits.

Typically : mounted outpost or road builders with the Scout + Sneak traits

FE V1.3 disclaimer : I thought SD had upgraded the "automatic road building system" between cities, and to Outposts and that road builders would become less important than before ... Finally, I'm not sure at all that anything has changed on automatic roads building, considering my current game.

No worry, on high difficulties AI starts with something like 1000-2500 gildar.

In my current game I ran out of gildar and couldn't make more Pioneers.  It seems like the Wealthy Sovereign trait is a required-take, now more than before.  The AI gets a lot of starting gildar at higher levels, but I'm not sure it knows what to do with it.  Every time I talk to them in the early game they've got 1500-2500 gildar.

There are some good ideas here, and thanks to Kael for explaining the difficulties.

I preferred it when pioneers cost population. It makes the population growth mechanic a lot more important. The difficulty stated is that the AI was playing badly with this. This seems an easy thing to fix? Just give the AI some rules about when to build a settler, that weighs up the advantages of doing so against the costs. I imagine a scaling factor that makes it very unlikely if the town is small, and quite likely if the town is very large. It isn't that complex a heuristic to build.

If the end result is that pioneer spam is reduced, that is a good thing.

I also like the idea of splitting out the tasks that a pioneer can perform into different units. Engineers to build outposts, pioneers for new settlements, etc.

Maybe a solution for settlers would be to require an influence cost. Sovereigns would need to have a small base influence income, and you could get one time influence bonuses for feats such as winning battles against the odds etc... in addition to existing mechanisms. And it would balance out a bit with henchmen: if you spend your influence on settlers, less to spend for henchmen.