One of the most progressive features in Civ IV was providing modders the ability to rewrite the AI to a large degree. The problem with the Civ IV AI API is that the AI developer is constrained to an AI design which is very similar to the original AI. Even with those constraints, modders have done some excellent things with it.
I'd like to see even more flexibility in modding the AI in Civ V. I'd like to see a public api which allows an AI to see the state of the game from the perspective of it's civilization analogous to the view the human player has, and allows the AI to manipulate its civilization in the same ways that a human can. This would allow modders to create their AI in any way they like. This interface can also guarantee that AI mods don't "cheat" which is a feature that many people clamor for.
Given such flexibility, I can think of any number of mods that I'd like to produce: a multi-threaded ai, a gpgpu ai, or even an ai which runs on a cluster. The sky's the limit, as long as the interface is agnostic of any specific AI design.
I agree with opening post.
I would propose an interface where you could connect AIs to Civ 5 thru TCP/IP. Where Civ 5 sends a state when game starts, then updates when changes occur. And the AI software can send commands back to move units, change settings in cities etc.
Thus the AI program contains all known info for the Civilization it plays and what it knows about its neighbors and environment. Leaving us fans free to develop that AI program as we wish, in what language we wish.
I would prefer a byte interface over XML or other highly readable but resource hog formats. But thats maybe just me.
There should be a default built-in AI in the game tho. And external AI as an option.