It's also letting me build a second Adventurer's Guild and a second Forge of the Overlord.

No, never lost my first city.  I only noticed this after capturing a couple of Magnar's cities and razing them.  I can't pinpoint exactly when it started though.

Yes, that could be, I did capture and raze his capitol.

Basically we should for each building add a cost to raze it... most of the time the cost is free, but for some buildings it will have a cost equivalent to the bonus received... allowing obviously for negative values in our resources.

I think this is the trigger. If you conquer and raze an enemy city with these buildings in them, it clears the flag and lets you build another.

The point is not costing you to raze enemy cities, but if you build an improvement that grants you +50 fame to build... then you demolish that building. You can build it again to gain another +50 fame. It is an exploit that needs to be addressed. Thus, i would say that if you demolish this particular building it should be -50 fame to your fame. If the operations to Raze a city are not tied to the razing a particular building, then what I suggest will avoid an obvious exploit, but if it is the same mechanism, so if you raze the entire city it is equivalent to razing each building associated with building costs, then I wouldn't suggest my solution to the obvious exploit.