3.1 Signal-to-noise ratio in Halo Doppler lidars

Halo Doppler lidars measure the noise level during periodic background checks, typically once an hour, in which the scanner is set to point to an internal (limited scan) or external target mounted on the instrument itself (hemispheric scan) so that no atmospheric signal is recorded. The raw signal from the amplifier during the background check (P_bkg) is saved as a profile in ASCII files (“Background_ddmmyy-HHMMSS.txt”) with the range resolution configured for the normal measurement mode. For most Stream Line and Stream Line Pro firmware versions, P_bkg is written on one line with a fixed precision of six decimals but a varying field width for each range gate. In Stream Line XR firmware, the P_bkg value at each range gate is written on its own line.

Figure 1(a) P_bkg measured by lidar 46 on 6 September 2016 at 23:00 UTC and P_bkg measured by lidar 146 on 1 May 2018 at 10:00 UTC. P_fit is also indicated for both systems. (b) 2-D histogram of mean P_bkg vs. T for lidar 46. (c) 2-D histogram of mean P_bkg vs. T for lidar 146.

Download

In most Stream Line and Stream Line Pro instruments, the profile P_bkg(z) is flat (constant with range) or presents a small linear increase with increasing distance z from the lidar (Fig. 1a); P_bkg(z) following a second-order polynomial can also occur (Manninen et al., 2016). For Stream Line XR instruments, P_bkg(z) can vary between a linear and an inverse exponential shape (Fig. 1a), for which the inverse exponential P_bkg(z) can be represented as

where b₁, b₂ and b₃ are scalars and can be determined from a least-squares fit.

In Stream Line and Stream Line Pro lidars, the magnitude of P_bkg increases non-linearly with instrument internal temperature (T) (Fig. 1b). For Stream Line XR lidars, which use a different amplifier, the mean P_bkg does not depend on T; however, the amplifier alternates randomly between a high mode ($P_{\mathrm{bkg}}\approx \mathrm{3.6}\times {\mathrm{10}}^{\mathrm{8}}$ for lidar 146) and a low mode ($P_{\mathrm{bkg}}\approx \mathrm{3.2}\times {\mathrm{10}}^{\mathrm{8}}$ for lidar 146) as seen in Fig. 1c. Furthermore, it appears that the inverse exponential P_bkg shape only occurs in the low mode, but not all low-mode P_bkg profiles follow Eq. (1).

The Halo Doppler lidar firmware accounts for changes in P_bkg level by calculating SNR as

where P₀(z) is the raw signal from the amplifier during each measurement, P_bkg(z) has been obtained during the previous background check and scalar scaling factors A₀ and A_bkg are determined online for each P₀ andP_bkg profile. Here, we denote the unprocessed SNR output by the instrument as SNR₀. Note that A₀ and A_bkg are not saved by the firmware, which means that the high and low mode in Stream Line XR lidars cannot be identified in the SNR₀ time series.

Equation (2) is straightforward to determine online as there are no assumptions about the shape of P_bkg, and it gives a reasonably good first estimate of SNR. However, Eq. (2) is vulnerable to inaccuracy in determining A₀ and A_bkg as well as to any deviation from the actual noise level during measurement of P_bkg. An offset in A₀ inflicts a constant offset in SNR₀ in a single profile, while an offset in A_bkg does the same for all profiles between two background checks (cf. Manninen et al., 2016). The magnitude of typical offsets in A₀ and A_bkg varies from instrument to instrument; in some cases they can have a major effect on data coverage (Manninen et al., 2016).

In all Halo Doppler lidars P_bkg contains a small but varying offset from the actual noise level at each range gate because of the finite duration of the background check. These offsets appear as a small constant offset in SNR₀ at each range gate between two background checks. To minimise the effect of offset in P_bkg, the integration time of background check measurement was originally designed to be 6 times as long as the integration time in measurement mode. The duration of the background check is user-configurable; however, for long integration times of up to 6 min considered in this paper such long background checks are not a viable option. In the next section we present an improved algorithm to correct SNR₀ for inaccuracies in A₀, A_bkg and P_bkg.

3.2 Improved SNR post-processing algorithm

Whether P_bkg(z) is linear or follows some other functional form is readily determined by fitting expected functions to it (cf. Manninen et al., 2016). For Stream Line and Stream Line Pro lidars we consider a second-order polynomial to represent P_bkg(z) better than a linear fit if it has at least 10 % lower root-mean-squared (rms) error than the linear fit to P_bkg(z). For Stream Line XR we consider Eq. (1) to represent P_bkg(z) better if it has at least 5 % lower rms error than the linear fit to P_bkg(z). Furthermore, knowing the typical noise level of a certain instrument, a rms threshold can be applied to discard bad fits and to flag periods of increased uncertainty.

Denoting the selected fit to P_bkg(z) as P_fit(z), the residual is

Averaging P_bkg,res(z) over a large number of P_bkg(z) profiles reveals a persistent structure in the residual (Fig. 2a). This part of P_bkg,res(z) originates in the amplifier response to the transmitted pulse, denoted here as P_amp(z), and it is the main reason for using the gate-by-gate defined P_bkg(z) profile in SNR calculation by the manufacturer. However, P_amp(z) stays reasonably constant over time and can be obtained from a long enough data set of P_bkg(z). Here, we used a discrete wavelet transform with a Symmlet order 8 wavelet as a low-pass filter to de-noise the averaged P_bkg,res(z). As shown in Fig. 2a, P_amp(z) is instrument-specific and needs to be determined individually for each device.

Figure 2(a) Lidar 46 P_bkg,res averaged from 20 August 2016 to 14 June 2017 (7193 background checks). Lidar 53 P_bkg,res averaged from 1 January 2014 to 30 November 2015 (16 802 background checks). P_amp is plotted for both systems. (b) First 100 range gates of lidar 46 P_amp calculated for different ranges of T. (c) Lidar 146 P_bkg,res averaged from 12 January to 31 May 2018. P_bkg,res is averaged separately for high P_bkg mode (1375 background checks) and for low P_bkg mode (1623 background checks). P_amp is plotted for both modes.

Download

In Stream Line and Stream Line Pro lidars, T has a small effect on P_amp(z) as seen in Fig. 2b. However, this can be addressed based on a suitably long T data set and P_bkg,res(z) by determining P_amp(z) as a function of the internal temperature. In practice, at least 300 P_bkg(z) profiles are required to obtain a reliable estimate of P_amp(z). Consequently, for an 11-month measurement campaign at Welgegund, we could determine P_amp as a function of T at 1 ^∘C resolution from 25 to 31 ^∘C (Fig. 2b). For T<23 ^∘C or T>35 ^∘C we could only determine aggregate P_amp profiles, but then these temperature ranges comprise only 9 % of the measurements in this data set. For optimal data quality, additional temperature stabilisation could be applied to ensure that P_amp is always in the well-characterised temperature range.

In Stream Line XR lidars, P_amp does not depend on T; however, P_amp has to be determined separately for the high and low mode of P_bkg (see Fig. 1c). For lidar 146, we define P_bkg high mode as mean $P_{\mathrm{bkg}}>\mathrm{3.4}\times {\mathrm{10}}^{\mathrm{8}}$ and P_bkg low mode as mean $P_{\mathrm{bkg}}<\mathrm{3.4}\times {\mathrm{10}}^{\mathrm{8}}$, respectively. As seen in Fig. 2c, P_amp for these two modes differs substantially.

We consider the sum of P_fit(z) and P_amp(z) as the best estimate for the actual instrumental noise level during a background check:

Using Eq. (2), we can move from a P_bkg-based SNR (i.e. SNR₀) to a P_noise-based, corrected SNR (denoted here as SNR₁) simply as

Next, we utilise the Manninen et al. (2016) algorithm to identify any possible bias in the A₀ to A_bkg ratio. In short, Manninen et al. (2016) cloud and aerosol screening is applied first to time series of SNR₁. Note that typically cloud and aerosol signal is easier to discern in SNR₁ than in SNR₀, and thus cloud screening is applied after Eq. (5). Then, first- and second-order polynomial fits are calculated for each cloud-screened profile of SNR₁(z), and a rms threshold is used to select the appropriate fit, similar to determining P_fit(z). Denoting the selected fit to cloud- and aerosol-free measurements as SNR_fit(z) we obtain

which is our final corrected SNR.

Note that to correct only for the bias in the A₀ to A_bkg ratio, a scalar denominator in Eq. (6) would be sufficient. However, using the fitted profile SNR_fit(z) as the denominator accounts for possible changes in the slope of P_noise since the last background check.

3.2.1 Implications for Stream Line XR lidars

The calculation of SNR₂ with Eq. (6) relies on the fitting to cloud- and aerosol-free measurements. For Stream Line and Stream Line Pro lidars, which do not exhibit the inverse exponential P_bkg shape, SNR_fit(z) will capture the shape of the actual noise level in nearly all cases. However, for Stream Line XR lidars the randomly occurring inverse exponential P_bkg(z) shape (Fig. 1a) is almost always masked by aerosol and/or cloud signal during measurement. Thus, it is not possible to correct for changes in shape of P_noise(z) with SNR_fit(z) during post-processing. However, the magnitude of uncertainty in P_noise(z) can be estimated from the average depth of the inverse exponential dip in P_bkg(z) during background checks (see Fig. 3a).

Figure 3(a) 2-D histogram of the ratio of P_noise(z) (best estimate) to $P_{\mathrm{noise}}^{\prime }\left(z\right)$ (linear fit only) for lidar 146 background checks. The mean of the ratio is also indicated. (b) σ_vr as a function of SNR for lidar 146.

Download

Now as A₀ is not saved, it is not possible to tell whether the amplifier was operating in high or low mode during measurement. Consequently, the difference in P_amp for the amplifier high and low modes also adds to the uncertainty in SNR for Stream Line XR systems, but, compared to the effect of the inverse exponential shape of P_bkg(z), the effect of P_amp is approximately 10 times smaller. However, any possible bias in the A₀ to A_bkg ratio can be corrected, and this is readily done by applying a linear fit to SNR₁(z) at range gates 100–400 (for which SNR is not affected by inverse exponential P_bkg) and using this as SNR_fit(z) in Eq. (6).

In practice, there are two options for SNR post-processing for Stream Line XR lidars. The first option is to accept the fitted Eq. (1) for P_fit(z) when it describes P_bkg(z) better. With this approach SNR₂ may overestimate the actual SNR if the shape of P_noise changes from Eq. (1) to being linear after the background check. Correspondingly, a change from a linear P_noise to the inverse exponential shape during measurement results in SNR₂ underestimating the actual SNR.

The second option for Stream Line XR SNR post-processing is to calculate a linear fit to P_bkg(z) based on range gates 100–400 and to always use this for P_fit(z). In this case, we only use high-mode P_amp(z) in calculating the noise level (Eq. 4) and denote it as $P_{\mathrm{noise}}^{\prime }\left(z\right)$. Consequently SNR^′₂ is the lower limit of the actual SNR, which can be useful if an SNR threshold is used to determine the usable signal for further analysis. For lidar 146 background checks from 12 January to 31 May 2018, the underestimation was on average 0.5 % of SNR+1 at the first usable range gate and decreased rapidly with increasing range (Fig. 3a). In the worst case, the underestimation at the first usable range gate was 3 % of SNR+1.

Uncertainty in SNR leads to uncertainty in σ_vr, as σ_vr is mostly a function of SNR (Pearson et al., 2009). However, σ_vr decreases rapidly with increasing SNR (Fig. 3b). Therefore, even the worst-case underestimation in SNR only has a limited effect on σ_vr if SNR is even moderately high (>0.03, −15.2 dB). On the other hand, for observations >2000 m away from the lidar, where signals are typically low, the uncertainty in SNR is also low (Fig. 3a). In the end, uncertainty in SNR and its effects in β and σ_vr need to be evaluated individually for each profile in Stream Line XR lidars.

4.1 Welgegund 6 September 2016

On 6 September 2016 lidar 46 was operating at Welgegund, South Africa, and a time series of SNR in vertically pointing measurement mode for this day is presented in Fig. 4. In this case, SNR₀ is very close to 0 when there are no clouds or aerosol present (Fig. 4a), indicating that the online calculation of A₀ and A_bkg is quite successful. However, the presence of small but varying offsets in P_bkg(z) is apparent in Fig. 4a as horizontal stripes in SNR₀ time series between the background checks conducted on the hour.

Figure 4Data from lidar 46 at Welgegund on 6 September 2016. (a) Time series of the SNR₀ profile in vertically pointing mode. (b) Time series of the SNR₁ profile in vertically pointing mode. (c) Time series of the SNR₂ profile in vertically pointing mode. (d) σ_SNR as a function of integration time per profile for SNR₀ and SNR₂ for range gates at 4800–9000 m a.g.l. Also $\mathit{\sigma }/\sqrt{N}$, where σ is σ_SNR at an integration time of 7 s (original integration time per profile) and N is the number of averaged profiles, is included in (d).

Download

In the SNR₁ time series (Fig. 4b) the horizontal stripes have been removed by applying the smooth P_noise-based background using Eq. (5). At the same time, the elevated aerosol layer at 2000–4000 m above ground level (a.g.l.) becomes easily discernible. Small biases in the A₀ to A_bkg ratio become visible as vertical stripes, for instance between 05:00 and 06:00 UTC in Fig. 4b, which are then corrected for in the time series of SNR₂ (Fig. 4c).

Comparing the standard deviation of SNR (σ_SNR) for cloud- and aerosol-free range gates shows a clear improvement in the noise level with the new post-processing algorithm (Fig. 4d). The main advantage of the new post-processing algorithm is that it enables averaging SNR; for SNR₀ any offsets in P_bkg(z) become the limiting factor. This is clearly seen in Fig. 4d, which shows σ_SNR for SNR₂ decreases with increasing integration time per profile following the $\mathit{\sigma }/\sqrt{N}$ rule closely as expected, but increasing integration time has little effect on σ_SNR for SNR₀.

Figure 5 demonstrates how the lower noise floor with the new post-processing algorithm allows vertical wind speed variance (${\mathit{\sigma }}_{\mathrm{w}}^{\mathrm{2}}$) up to 2000 m a.g.l. (i.e. up to the top of the mixed layer) to be determined on this day. Furthermore, by averaging the originally 7 s data to 168 s integration time per profile and applying the new post-processing algorithm, the SNR threshold at the 3σ level (see Fig. 4d) can be decreased from 0.0032 (−25 dB) for SNR₀ to 0.00065 (−32 dB) for SNR₂. Consequently, β can be retrieved for the elevated aerosol layer at 2000–4000 m a.g.l. (Fig. 5c, d). Note that the offsets in P_bkg(z) result in horizontal stripes in the 168 s integration time β calculated from SNR₀ in Fig. 5c.

Figure 5Data from lidar 46 at Welgegund on 6 September 2016. (a) Time series of the ${\mathit{\sigma }}_{\mathrm{w}}^{\mathrm{2}}$ profile, for which a threshold of 0.0031 (2σ) has been applied to SNR₀. (b) Time series of the ${\mathit{\sigma }}_{\mathrm{w}}^{\mathrm{2}}$ profile, for which a threshold of 0.0021 (2σ) has been applied to SNR₂. In (a) and (b) the instrumental noise contribution to ${\mathit{\sigma }}_{\mathrm{w}}^{\mathrm{2}}$ has been subtracted. (c) Time series of β obtained with 168s integration time from SNR₀; β has been filtered with a threshold of 0.0032 (3σ) applied to SNR₀. (d) Time series of β obtained with 168 s integration time from SNR₂; β has been filtered with a threshold of 0.00065 (3σ) applied to SNR₂. Mixing layer height (MLH) determined from panels (b) and (d) is also indicated.

Download

A lower noise floor also enables wind retrievals with a lower SNR threshold, which increases the data availability. The effect on data availability depends on atmospheric conditions, though. In this case for instance (Welgegund, 6 September 2016), a 75^∘ elevation angle velocity azimuth display (VAD) scan was utilised for horizontal wind retrieval every 15 min. With the new post-processing algorithm, the SNR threshold for wind retrieval could be decreased from 0.0045 to 0.0032. This decrease in the SNR threshold enabled the wind retrieval for 2–13 range gates more from each VAD scan; on average, winds could be determined from 7.5 additional range gates per VAD scan. That is, vertical coverage of wind retrievals increased on average by 200 m with the new post-processing.

Wind retrievals at lower SNR will have higher uncertainty due to higher instrumental noise in radial velocity measurement; yet enhanced SNR will enable more accurate determination of the instrumental uncertainty in wind retrievals. However, as a major fraction of the uncertainty in retrieved winds arises in atmospheric turbulence (Newsom et al., 2017), the more accurate SNR will only have a limited effect on the overall uncertainty in the wind retrieval. Therefore, the uncertainty in each wind retrieval should be evaluated, e.g. with the methodology of Newsom et al. (2017), before the wind retrievals are disseminated.

4.2 Helsinki 1 and 6 May 2018

Measurements using lidar 146 at Helsinki, Finland, on 6 May 2018 (Fig. 6) present all the issues with a Stream Line XR lidar at its worst. In Fig. 6a, SNR₀ is negative, e.g. from 01:00 to 05:00, 06:00 to 10:00 and 12:00 to 13:00 UTC because of an erroneous A_bkg coefficient. On the other hand, the individual profiles with unrealistically high SNR₀ around 11:00, 14:00 to 15:00 and 20:00 to 21:00 UTC indicate errors in the A₀ coefficient. Additionally, horizontal stripes in SNR₀ time series similar to lidar 46 (Fig. 4a) indicate offsets in P_bkg(z). The reason for poor determination of A₀ and A_bkg for lidar 146 seems to be that P_bkg(z) is frequently non-linear, unlike for lidar 46, for example.

Figure 6Data from lidar 146 at Helsinki on 6 May 2018. (a) Time series of the SNR₀ profile in vertically pointing mode. (b) Time series of the SNR₁ profile in vertically pointing mode. (c) Time series of the SNR₂ profile in vertically pointing mode. (d) Time series of the SNR^′₂ (always based on a linear fit to P_bkg(z)) profile in vertically pointing mode.

Download

The new post-processing algorithm corrects the errors in A₀ and A_bkg as well as the stripes due to offsets in P_bkg(z) as seen in Fig. 6c. However, Fig. 6c shows that P_bkg(z) changing between the inverse exponential and linear shape causes over- and underestimation of SNR₂ in the lowest 1500 m a.g.l. For instance, positive SNR₂ in the lowest 1000 m at 12:00 to 13:00 UTC and negative SNR₂ in the lowest 1000 m at 14:00 to 15:00 UTC are due to noise level shape changes between background check and measurement modes. During these periods, the lidar signal is fully attenuated by a cloud within the lowest 200 m, and consequently SNR₂ in the 200–1000 m range should be zero. In Fig. 6d, SNR^′₂ is only calculated using a linear fit to P_bkg(z) as discussed in Sect. 3.2.1. This removes the overestimate of SNR at 12:00–13:00 UTC, but cannot correct the underestimates.

Measurements with lidar 146 on 1 May 2018 at Helsinki present much less noisy SNR₀ than on 6 May 2018, as seen in Fig. 7a. On this day there are cirrus clouds present at 8000–12 000 m a.g.l., but the stripes due to offsets in P_bkg(z) make it difficult to distinguish the clouds from noise in SNR₀. Applying the new post-processing algorithm and increasing integration time from 10 to 60 s for this day enables the SNR threshold at the 3σ level to be lowered from 0.0035 (−24.5 dB) for SNR₀ to 0.0012 (−29 dB) for SNR₂. This results in a significant increase in data coverage for the cirrus clouds, as shown in Fig. 7c and d.

Figure 7Data from lidar 146 at Helsinki on 1 May 2018. (a) Time series of the SNR₀ profile in vertically pointing mode. (b) Time series of the SNR₂ profile in vertically pointing mode. (c) Time series of β obtained with 60 s integration time from SNR₀; β has been filtered with a threshold of 0.0035 (3σ) applied to SNR₀. (d) Time series of β obtained with 60 s integration time from SNR₂; β has been filtered with a threshold of 0.0012 (3σ) applied to SNR₂.

Download

4.3 Finokalia 8 July 2014

Time series of SNR in vertically pointing mode with lidar 53 on 8 July 2014 at Finokalia, Greece, are presented in Fig. 8. On this day, SNR₀ is close to 0 for 4000–9600 m a.g.l. elevation (Fig. 8a), indicating that the online calculation of A₀ and A_bkg is quite successful. Only at 00:00–01:00 and 20:00–21:00 UTC is SNR₀ negative, indicating a small offset in A_bkg. However, horizontal stripes in the SNR₀ time series between the background checks are apparent in Fig. 8a, indicating the presence of small but varying offsets in P_bkg(z).

Figure 8Data from lidar 53 at Finokalia on 8 July 2014. (a) Time series of the SNR₀ profile in vertically pointing mode. (b) Time series of the SNR₂ profile in vertically pointing mode. (c) Time series of β obtained with 350 s integration time from SNR₀; β has been filtered with a threshold of 0.0044 (3σ) applied to SNR₀. (d) Time series of β obtained with 350 s integration time from SNR₂; β has been filtered with a threshold of 0.00059 (3σ) applied to SNR₂.

Download

After SNR post-processing (Fig. 8b), elevated aerosol layers at 1000–4000 m a.g.l. are clearly visible on this day. These aerosol layers were also observed with a co-located multiwavelength Raman lidar, Polly XT (Baars et al., 2016; Engelman et al., 2016). A comparison of lidar 53 and the Raman lidar measurements at 1064 nm wavelength is presented in Fig. 9. Considering the wavelength difference, the agreement between the two systems is reasonably good. Further averaging of SNR₂, in this case up to 350 s integration time, allows the determination of β for the elevated aerosol layers. With this long integration time we can reach a 3σ SNR threshold of 0.00059 (−32 dB) for SNR₂. For SNR₀, offsets in P_bkg(z) are the limiting factor in determining the SNR threshold, and at the 3σ level only 0.0044 (−24 dB) can be achieved.

Figure 9(a) Vertical profiles of SNR from PollyXT at 1064 nm wavelength and SNR₂ from lidar 53 at Finokalia on 8 July 2014. Both profiles are obtained at 21:00 UTC; the integration time of the lidar 53 profile is 350 s, and the integration time of the PollyXT profile is 360 s. (b) Time series of PollyXT SNR at 1064 nm wavelength with 360 s integration time at Finokalia on 8 July 2014. (c) Time series of PollyXT attenuated backscatter at 1064 nm wavelength with 360 s integration time at Finokalia on 8 July 2014.

Download