I hope/expect that the current flat-rate penalty is just a first approximation. Other games either give you was way to assimilate the populations of captured cities (Rome Total-War) or have more or less of a penalty depending on what the different races think of each other (MoM) or even have you replace the conquered population at the start (MoO & MoO2, iirc) or replace them later (AoW:SM). There are a lot of ways, but I don't think I would like a permanent fixed penalty, and doing in the original population might be too politically incorrect, even for a fantasy game.
So maybe one or more of the following..
Let you send pioneers to settle there reduce unrest in proportion to the percent of resulting population from your race.
Have less antipathy between Kingdoms and between Empires than there is between Kingdoms & Empires.
Have antipathy automagically decline over time.