I feel like this is a case where 'keep it simple' might be better.
Having too complex of a visual range I feel would confuse early users and create an 'elite' group of players that actually understand its mechanics.
I'd keep standard 'sight' range (with things like scouts obviously seeing further than normal bums)
Then I'd suggest several abilities or modifiers. I'd suggest a few forms of this.
"stealthy" - that would reduce the distance that a unit can be spotted by 1 or 2 spaces, to a minimum of 2 spaces (example: a watch tower that could see... say 6 spaces, could only see them at 4). I'd think several variations of this would be possible, including 'camoflaged' which might reduce by more than 'steathy' and 'sneaky' which might reduce by less)
"invisible" - which would prevent any detection unless at 1 space
"true invisible" - prevents any overland detection at all except by those able to see through invisability or stealthy abilities. To normal units it would only be visable in tactical battle.
Then simply there would be a few counter abilities.
observant: reduces invisibility by 1 space (allows invisable units to be seen at 2 spaces. No effect on truely invisible)
alternate vision: (seeing with sound, infared, or perhaps slightly magical perception) removes invisibility, reduces a space or two from 'stealthy' varients, makes true invisibility act as a normal invisibility: can be seen at 1 or 2 space on overland map as well as tactical map)
see invisibility: is straight-up able to see invisible units. maybe not stealthy varients
true-sight > nothing gets past these guys... no way to reduce, shroud (I imagine darkness spells that would obscure unit totals and ability to observe cities), or otherwise hinder the vision of these guys.
And in terms of terrain, Forests should obscure vision to 1 spaces, modified by things like alternate vision and observant the same way as camoflage or stealthy would.
I don't think there should anything complex like a formula for making 1 man less visable than 10000 men (I'd make the 1 man with 'stealthy' unable to be detected within the army, where the 10000 'generic soldiers' would be apparent) or making flying units different from ground units (I might give them 'observant' in the case of an eagle or a hawk) or giving them additional stats beyond the basic 'line-o-sight' stat. No need to make mathmatics complex. (*wonders if what I consider as 'simple' is actually complex*).
I there there is a bit of strategy there too. If you threw in 2000 invisable warriors and 2 phantom beasts (remember fantastic monsters are a 'big deal') , then surrounded them with 1000 generic soldiers. Somebody might look at the screen and be like "oh, my upgraded hero can take on those 1000 soldiers no problem. He's riding a freaking bear!" and then when he enters battle he's stormed by a bunch of previously undetectable units that were hiding with the 1000 soldiers