I get this issue as well; without getting even close to Fortresses at times.
Last game it happened very early, and seemed to simply be related to zooming out (undrawing detail) to cloth map, then upon zooming back into the 3D engine it didn't readd the detail.
Often in place of buildings I'd get a smoke stack showing and nothin' else. 
