Sadly in a world with things that can fly, burrow, teleport, become incorporeal, transform earth to mud, disintegrate, breath fire, etc castles and all types of fortifications we are used to see in our world are totally useless.
Castles are nice and make sense in a game like Civilization, but in a world with those things defensive structures and tactics will have probably evolved pretty different than in our world (an old Dragon Magazine had a great article about this subject).
As Scoutdog mentions there should be defensive spells and wards as well... you even named one of them = earth to mud. The spell earth to mud is a defenitely a spell to prevent an advancing army from reaching gates so easily.
In regards to flying creatures a good defensive castle should have ballistas, magic domes, and mages ready to cast something like Double Gravity as seen from AoW:SM.
In regards to burrowing... I like this game idea however it's highly unlikely we'll see burrowing being used on the battlefield or world map. Even in the rare chance it is available only a few limited units would have such a powerful ability.
In regards to disintegrate it would have to be a very very powerful mage to disintegrate an entire city wall... so powerful it's something I would only give to a powerful independent, such as an angel, devil, etc., . Also in regards to city walls they can be enchanted by ritual spells several weeks before a battle occurs... these spells may take longer to cast, but can provide extra strength and resistance for the wall.
Well I could continue.... but the point being that within a fantasy world there are actually more ways to explain defensive bonuses than less ways when compared to real world history.