When something finishes in production there is a little sound effect that is played and it will show up as an event on the event list on the right side of the screen.  Also, as frij pointed out, the city will get Zzz's on the empire tree (on the left side of the screen) when it's idle.  

If the event list and sound aren't being apparent enough to the majority of players I will talk to Derek and see what can be done to improve them.

I like how it is now, not too intrusive. You could put a selectable option for players who want more so that players get a pop up when construction is complete. Just an idea.

Actually there are a few improvements that provide like extra gildar or research ect. if the production que is empty. Though I think all those are special unlocks from city level ups.

My cities constantly end up with no production queues because I miss the notification of reaching the end of production queue.

Soldiers have maintenance as well. If you build too many, just for the sake of building them, you will have less of a treasury when needed at best, or run out of money at worst.

I found this out eventually. It was mostly the early game where this bothered me. I think an optional pop-up would be optimal. I would want it for the early game, but it would probably be more of a nuisance late-game on a large map.

While I am quite used to the WoM way of showing idle cities, I am totally prepared to learn a new way. I noticed that my cities get yellow and sleepy on the left when they are not producing. That is enough for me.