Leica AT40x: Refractive index errors

I have discovered some issues with instrument Leica AT401. Most likely, the issues concern Leica AT402 as well. The problems are related to calculation of refractive index of air with firmware 1.x and 2.x until now. The problems may be solved by Leica in a future firmware version as I have reported all that to Leica Geosystem in Switzerland.

1) Old refractive formulae are used by default for calculating refractive index of air. Modified Edlén (1966) formulas are used. They differs up to approx. 0.6 ppm from precise procedure given by Ciddor and Hill (1996, 1999, 2002). C&H procedure is approved by IAG.

2) Wrong effective carrier wavelength is used in computation of refractive index. Instead of value 780nm, 795nm should be used. It causes an error of approx. 0,3ppm.

3) Value of refractive index of air is not updated in the instrument´s memory until it differs on the 6th decimal place from an old value. Even if an user put in new value manually, it is not updated. Therefore an maximum error of 0.5e-6 in refractive index and approx. 0.5 ppm in measured distance occurs.

All these errors may seem negligible for standard geodetic total stations, but they are not for absolute trackers because accuracy of the instruments given by manufacturer is exceeded. Fortunately, clever user may avoid and eliminate these errors while programing an user-designed application. Let me know if you would like to know more details.

Warning to all users: Until today, more than a year after Leica was notified about the problem of improper updating of refractive indices, the issue has not been fixed so far. It is still present in the current firmware 2.2.0.5979 (2014-08-20).