I have seen this happening to campsites, mines and shards as well. At least on the mines and shards this occured for me in the context of destroying the building built by an opponent on that site. In such cases the object reappeared a few rounds later, in case of inn and campsite this self-repair likelyhood seemed to be lower.
are you pillaging enemy territory ? in this case the rubble is smoldering and the AI rebuilds everything the following turn
pillaging seems to be a waste of time, you don't earn cash and the AI keeps rebuilding, there is no point, if at least it was useful to empty their gold reserve ...