Ok, you clearly gave your ideas quite a bit of thought, so I feel it fair to address them individually.
Diplomacy is one area that has been severely neglected in all of gaming. The only game I've ever played that had good diplomacy was an MMO called Shadowbane (and the diplomacy occurred between real players and involved the real consequence of losing your city).
The reason you only see this with real players is the same reason that you are going to see me using alot to explain some of the shortcomings of diplomacy AI in the past: it is simply impossible to create an AI that thinks like you and I do. The very concept is something that devoted theoretical researchers quest unsuccessfully for. AI will not think like a human, not even close, for a few decades (at least). Its not generally "neglected" so much as it gets to a point where the investment of time into AI behavior becomes astronomical, and the returns diminish drastically.
Metrics are totally the wrong way of doing diplomacy. You can't quantify a real person's opinions or a culture's customs with some silly numbers. You can't quantify hidden agendas or secret agreements.
The AI simply can't have "oppinions" or "hidden agendas". Perhaps the AI could be coded to look for a valuable artifact, and may even be capable of tricking the player into thinking that the AI isn't looking for it, but as for true subterfuge... we wont see that for a long time. Metrics could be hidden, and thier could be some random variables at work, along with a script written to give the AI some initiative, but metrics are the only way to reliably represent a changing oppinion in computer language.
A few key rules should be followed when designing the diplomacy system of a game. The AI-controlled players should:
1 Never give away any information unless it suits their agenda to do so
2 Never make their decisions in a random or otherwise non-deterministic way
3 Never treat the human player(s) differently from how they'd treat their AI rivals
4 Always seek to maximize their position in the course of negotiations
All of these rules have been broken by games in the past, to their detriment.
I numbered your suggestions so I could counter them one at a time.
1- As I said above, its "agenda" will likely be to survive, and little more. Players may also get mightily annoyed if the AI is unwilling to even hint at force makeup and strength - why be diplomatic with an empire you have no understanding of? The "mystery" quickly wanes and the lack of info will make players feel disconected.
2- Random may be the ONLY way to create an AI that acts with initiative. The sheer volume of variables that come into play in a TBS usually means that to take them all into account would cause paralysis amongst the AI. The AI shouldn't be stupid, but it will be pretty boring if its always doing the same thing with little variation
3- Agreed. Although the AI's are generally weaker than the human a short way into the game, so the human HAS to be treated like the big bad steamroller he is. But all things equal, so should diplomacy be.
4- Another good point, but the AI will rarely know whats "best" for it. 35,000 gold might seem like a great deal for a city, but what if its thier only Iron source? Or if it will cut the empire in two? Many games go with the strategy of undervaluing everything the human offers, and drastically over-valueing thier own position. This leads to stale diplomacy, but the alternative is generally human abuse. Hmm..
Those ideas are great, and game developers didn't just ignore the concepts in favor of something else, the concepts were simply unnatainable in a realistic manner. AI often still has to cheat just to face the human on the regular map, do you really think the subtleties of diplomacy are an easy (or even possible) thing to code?
Obviously, the design of a diplomacy system is much more complicated than following a few rules. There must also be a strong strategic foundation for the AI which allows it to formulate realistic goals and take the necessary steps to achieve them. AIs should be able to "see through" a player's gestures towards them and find weaknesses to exploit.
Ok, here is where you begin to lose me. If you want that kind of play, go online. The AI is there to provide a reasonable barrier for the human to beat up on, and not much else. The harder modes generally don't give better AI in most games, they give the AI bonus's. That is because the AI on normal (sometimes on easy!) is the best "thinking" AI you will get.
Tell me how you expect the AI to see through a human plan? Ill use a good example that has hounded some of my favorite games in the past...
A player masses troops on the AI border. What should the AI do? Now, your first (reasonable) reaction is "build a huge army! Prepare to defend!" and thats what many of us would do, but its going to break the AI quickly. Why? The AI suddenly shifts focus from growing to massing an army. If the human is about to invade, great! What if the border just happens to be where the human likes gathering his forces? Even if it is for an innevitable invasion, that invasion could be many, many turns away.
Should the AI crap its pants and abandon the expansion of its nation for an attack that may never materialize? Should the AI strike pre-emptively, thus throwing diplomacy out the door (and becoming the aggressor in what could be a nuetral situation)? Should it ignore the human, and risk being over-run? Should it even bother noticing the build up, perhaps the human moves forces through there all the time? Should it try to be peaceful with a human, who we all know will just abuse its hospitality and stab it in the back? Should it try forming an alliance, giveing gold and resources to other nations and weakening itself for what could be no real issue?
This is the example of a minor issue in a game with thousands of variables. Seeing through the human vale is impossible, even for us humans most of the time, why should the AI be any different?
If a player is weaker militarily than the AI but stronger economically, it should not allow the player to bribe it into abandoning its plans for invasion. Rather, it should attempt to suck as much gold, mana, spells and other bribes out of the player that it can to further bolster its position.
Good point actually, but the human will quickly wise up to the AI if it always acts this way. Why give away gold if the AI will just stab you in the back all the time? You already said you don't like random variables, so how will this work? Did you ever play civ4? after playing a game with monty or ghengis would you ever even talk to them again lol? I would hope we could avoid that predictability...
This type of system is very hard to design, I admit, but I believe it is worth trying. I hope that the scripts and routines used for the AI are open to the modding community so that modders can fix any glaring mistakes or loopholes in the diplomatic AI.
What are your suggestions to improve the diplomacy of this game and elevate it over the mistakes of the past?
It is infact so hard to design that it borders on being impossible. The whole thing could be done through scripting, but your looking at adding another year to productiong and having a massive product with AI's that take half hour turns on a good processor. Your ideas are impressive, but for the most part too pie-in-the-sky. The reason you have never seen it done is because it really cant be... and I have more faith in stardock than anyone when it comes to AI.
So, what do I think they should do? Leave it open and moddable, like you said. Throw some reasonable routines in so that the AI can be proactive. Make the AI realise glaring advantages, evaluate its situation, and act if possible. Powerful mana crystal right across the border? How powerful is the empire guarding it, if the AI is notably more powerful, go for it.
Also, have many AI personalities. One could be more aggressive, one more economic, etc. Make it possible to switch from economic to militaristic when needs arise, and back when the emergency ends. Have the AI also have a very long ranged "focus" ie magic, tech, numbers, money, etc, that it focus's on. Make it unpredictable and dynamic. The only time an AI ever beat me in a game was when it took me by surprise, while I was in my confort zone. Thats the way it needs to act, otherwise the game is a forgone conclusion 1/8th the way into it.