Yes, it takes a very very long time. Especially so, if you direct caravans as to maximize food production and not just to the next city at hand. Then the distances for one trip can be quite long and the road will effectively NEVER upgrade.
It rather should be something like an upgrade needing ... lets say 100 points and you get one point per turn on a road between city X and Y for every caravan that uses this road. In this way, heavenly frequented roads around your main caravan destination cities / city would upgrade faster than the periphery of your empire. And this would in my opinion feel more naturally than just having an incentive to make the round-trip or the caravan distance as short as possible.
As the system stands right now you can only choose between either food production efficiency or the still VERY long-term prospect of a road upgrade. In my opinion not much of a choice.