the prestige penalty is the more towns the slower they grow, and if you get 10 or MORE towns they typically grow at .1 per turn. or TEN turns per pop nomatter what the excess food state is
unfortunately all the workarounds for it will require starting a new game, either making a modified copy of the ElementalDefs.xml or creating accessories(eg rings) that add prestige( I have done both, the modified elementaldefs is needed to increase the number of towns before the growth hits almost ZERO) and the accessories are equiped onto heros that are stationed in each town that you want to grow.
if you want to read a bit more about the issue and workarounds you can read the prestige and woes thread.
Thanks for that!
Edit: i have given this issue some thought and i wondered if it would be worth building the cheapest possible units in less developed cities and then disbanding those units in the cities i need to advance? Or would i be better off razing those less developed cities instead?
I noticed that when i disband caravans, it gives me a report that some unit joined such and such a city? but when i build caravans, it does not seem to report any population usage? Could i constantly build caravans then disband them to boost city population growth?