Out of curiousity do you have any of these quests active when you use these different quests? Or are you clear of all quests at the time?

Yeah, likewise when it has happened to me - either no quests in log or only wildlands/master quest ones, and it has still happened.

I've had this too.  About 10-20% of the time the quest tile is unusable.  This is even when I have no quests, not even wildland quests (so it is not the case of the quest tile's quest already being active somewhere else).  Devs, please fix!  I know I have been a bit overeager in the past, but this is an important polish aspect with Quest Maps, so I think it would be very worthwhile to fix.

I have put forward savegame with this problem before. If you play as Altar and buy some quests you will get this every time (at least I do).