Would be easier and better to figure out how to give Prestige a reason to exist in end-game (% bonus to city production per point of prestige?) rather than figuring out how to make demolishing large numbers of building quicker.

As for housing, I don't particularly share your view of the problem. It's true that only x number of people are needed to level up a city, but when you recruit armies you use people from the population pool, so this kind of min/maxing is usually detrimental since you'll leave yourself with a smaller pool of people to recruit from. As a smaller side effect, if you do destroy all your Prestige buildings, then when you recruit a thousand guys for your armies, your population will crawl back that much slower

That's if I'm doing a conquest victory and churning out units as fast as I can.  If I'm pursuing a tech victory or a spell of mastery, then I have every reason to pick through my level 5 cities and weed out prestige generators and extra housing if I can replace them with research or spellpower buildings.

I agree that prestige mean something to level 5 cities.  Maybe there's emigration and people start filling up your small cities faster?  Maybe it provides a production bonus?

As for housing, what if you had a city governor who would convert slums <-> regular housing <-> mansions based on your orders?