When you scroll away from your sovreign, the destination marker vanishes as well.
Pretty easy to reproduce, set a destination somewhere away from your sovreign and then scroll away from him. Around the time he leaves the map (being obscured by the UI doesn't do) your destination will cease rendering. Scroll back so both are in view again, and it will show up again. Magic!
