Cities should be damaged when they are conquered, not just lose their population. The mechanic could strip off the most recently built buildings (or just those around the outside). The number of buildings destroyed could be commensurate with the fierceness of the battle (measured in terms of total HP lost).
Alternatively, the buildings could just become "damaged" and require time/money to repair. While repairs were underway you would not be able to recruit or build new buildings.
