So under the research tree, first you get a fort, then on a subsequent tech, you get a fortress. The fort comes with a palisade wall around your entire city, and the fortress comes with a stone wall. In my current game, I built a fortress first in the city, and my walls upgraded to stone. I then built a fort in the same city, and all my walls downgraded to a fort. I'm not sure what's up with this seemingly graphical-only effect, but thought you guys should know.