There's two things that may explain what you're seeing.
1. The mites may have left because the lairs were, for a considerable period of time, under the dominion of the player or an AI.
2. An AI army may have killed the mites, but failed to loot the lair, either because it lacked a champion, or because its champion thought there was something more important to do.