It seems to me that the concept of "specialists" is a more flexible way of giving building slots to cities.
instead of say,
lv 0 outpost - 1 building slot
lv 1 city - 3 building slots
lv 2 city - 6 building slots
lv 3 city - 10 building slots
lv 4 city - 15 building slots
lv 5 city - 21 building slots
combined with a building upgrade system, example: lumberjack gives 1 matt, upgrades to mill gives 4 matt, upgrades to "guild" gives 10 matt etc
slow down population growth a bit. 1 per turn is too fast in the beginning.
prestige can increase this rate, higher city level can increase it too.
pioneers could cost about 3 population, this would encourage the choice of using that population as a resource for something else, such as a building or levelling up your town.
this also maintains the potential to turn your outpost into a special type of fortification or garrison building specialized to training units - this would help stop spam as well by giving more choices into troop training (could train faster than towns do for example)
there are lots of options, you don't have to rewrite the whole system because something does not quite work the way you intended it.