Well, when we get strong enough armor it matters, and when you're not fighting a sovereign it matters. The reason why is how the combat system works:
In order to deal damage, the attack roll has to beat the defense roll. A unit with enough defense can actually be flat out immune to attacks from weaker units. Right now we aren't really scaling up enough to get to that point, and the units we have do minimal damage to the really strong stuff on the map. But as we get more tech and more training (and enchanted weapons), you'll see elite units that are more of a force.
That said, I do think the veteran and elite training options need to add more HP then they do.
Fundamentally, normal units will never be super effective against really big things. But trained and geared units will do better against peasant hordes.
