2.1.1 Initial lidar ratios for aerosols

The initial lidar ratios and their uncertainties for several of the aerosol subtypes have been revised for V4 (Kim et al., 2018). As can be seen in Table 1, V4 specifies larger 532 nm lidar ratios for dust (by 10 %), clean marine (by 15 %), and clean continental (by 51 %) aerosols. These higher lidar ratios, which are consistent with improved knowledge gained over the past decade (e.g., Tesche et al., 2009; Nisantzi et al., 2015; Liu et al., 2015; Papagiannopoulos et al., 2016; Haarig et al., 2017), contribute to higher values in the retrieved backscatter, extinction, and optical depths than were reported in V3. Our better understanding of lidar ratio variability led to significant reductions in the uncertainties ascribed to the lidar ratios for marine, desert dust and smoke aerosols, resulting in generally lower uncertainties in the retrieved products for these aerosol types. As explained in Kim et al. (2018) and documented in the V4 CALIPSO Data Products Catalog (Vaughan et al., 2018c), the nomenclatures of the aerosol type 3 “polluted continental” and type 6 “smoke” were changed in version 4.1 to “polluted continental/smoke” and “elevated smoke”, respectively. V4 also adds a new “dusty marine” tropospheric subtype and five new stratospheric subtypes. The lidar ratio for the dusty marine mixture is significantly higher than that for clean marine, but significantly lower than that for polluted dust. Aerosols identified as dusty marine in V4 were generally classified as polluted dust in V3. Some aerosol lidar ratios at 1064 nm have also been changed, as shown in Table 1. Note that there was no cloud–aerosol classification in the stratosphere in V3 and that all stratospheric layers were assigned an initial lidar ratio of 25 sr and a multiple scattering factor of 1.

Combined with changes in the CAD and aerosol subtyping algorithms that have led to changes in the identification of some aerosol types, these new lidar ratios yield extinction profiles and optical depths that can differ noticeably from those reported in V3. For aerosols, changes to optical depths and extinction profiles are largely the result of changes to the lidar ratios for the particular aerosol subtypes, and of changes in the algorithms that determine these subtypes, and are only rarely due to changes to the extinction algorithm. Therefore, changes in aerosol optical depths are discussed in a separate publication (Kim et al., 2018).

Table 1Initial lidar ratios and their uncertainties, in units of steradians, that characterize each of the aerosol subtypes reported in V3 and V4. V4 values that have changed with respect to V3 are in bold text.

n/a: not applicable.

Download Print Version | Download XLSX

2.1.2 Multiple scattering factors and initial lidar ratios for clouds

Ice clouds

In contrast to V3 and earlier data releases, where the multiple scattering factors for ice clouds were assigned a constant value of η=0.6, ice cloud multiple scattering factors in V4 are approximated by a sigmoid function of the temperature at the centroid of the 532 nm attenuated backscatter coefficients within the cloud. The sigmoid function used in the V4 retrievals was derived by reconciling three years (2008, 2010, 2013) of CALIOP effective two-way transmittance measurements $\left(T_{\mathrm{CALIOP}}^{\mathrm{2}}=\exp \left(-\mathrm{2}\mathit{\eta }{\mathit{\tau }}_{\mathrm{CALIOP}}\right)\right)$ in semi-transparent clouds with the collocated absorption optical depths (τ_IIR) retrieved by the CALIPSO Infrared Imaging Radiometer (IIR) at 12.05 µm (Garnier et al., 2015). This analysis used only clouds measured at night and identified with high confidence as being composed of randomly oriented ice (ROI) (Hu et al., 2009). Furthermore, to guarantee the highest quality retrievals of τ_IIR, it was restricted to single-layer clouds measured over oceans. As seen in Fig. 8a in Garnier et al. (2015), the effective optical depths (i.e., ητ_CALIOP) obtained from direct measurements of $T_{\mathrm{CALIOP}}^{\mathrm{2}}$ must be rescaled by a temperature-dependent multiple scattering factor to ensure the appropriate temperature-dependent correspondence between τ_CALIOP and τ_IIR. As shown in Fig. 1a, the V4 sigmoid approximation of η, which was retrieved assuming that these clouds are composed of severely roughened aggregated columns (Yang et al., 2013), increases from a value of 0.46 at a centroid temperature of 0 ^∘C to 0.79 at −90 ^∘C.

As a consequence of the changes in the η parameterization, the initial lidar ratio estimates for ice clouds have also been changed. In V3 and earlier data releases, an initial lidar ratio of 25 sr was used in all ice cloud extinction retrievals. Comparisons of V3 optical depths for semi-transparent ice clouds with IIR absorption optical depths at 12.05 µm (Garnier et al., 2015) and a radiative closure experiment using MODIS 11 µm radiances by Holz et al. (2016) showed, on average, quite good agreement for constrained retrievals. However, for unconstrained retrievals, several studies of semi-transparent ice clouds showed that, when used in conjunction with η≈0.6, the initial lidar ratio of 25 sr was generally too small. Josset et al. (2012) found, from their analyses of optical depths determined from CALIOP and CloudSat echoes from the ocean's surface, that the initial lidar ratio was about 25 % too low. Their analysis of semi-transparent cirrus produced a mean multiple scattering factor of 0.61±0.15, with a corresponding mean lidar ratio of 33±5 sr that showed slight variations as a function of temperature and latitude. A similar conclusion was reached by Garnier et al. (2012), who compared CALIOP optical depths with those measured by the IIR at 12.05 µm and found that there was an inconsistency between CALIOP V3 constrained and unconstrained optical depths, and that using a revised initial lidar ratio derived from constrained retrievals greatly improved the comparisons. Holz et al. (2016) showed that the CALIOP unconstrained optical depths were substantially smaller than the MODIS Collection 5 values. The authors showed that a much better agreement was achieved when CALIOP unconstrained retrievals using a lidar ratio of 32 sr and a multiple scattering factor of 0.6 were compared with the MODIS Collection 6 (C6) optical depths.

To ensure consistency between unconstrained and constrained retrievals in semi-transparent ice clouds, the V4 initial lidar ratios have been derived from a statistical analysis of nighttime constrained retrievals in clouds identified with high confidence as being composed of ROI. These constrained retrievals used direct measurements of effective two-way transmittance, together with multiple scattering factors generated by the aforementioned sigmoid approximation function, to retrieve optimized estimates of the V4 initial cirrus cloud lidar ratios. As seen in Fig. 1, the resulting initial cirrus lidar ratios and the corresponding layer-effective lidar ratios (i.e., the product of lidar ratio and multiple scattering factor) both vary as functions of the layer attenuated backscatter centroid temperature. Like the multiple scattering factors, the initial cirrus lidar ratios (Fig. 1b) are approximated by a sigmoid function of backscatter centroid temperature, and decrease from a value of ∼35 sr to ∼21 sr as the layer centroid temperature decreases from 0 to −90 ^∘C. It can be seen that the V4 effective lidar ratios (Fig. 1c) are greater than the V3 value ($\mathrm{25}\times \mathrm{0.6}=\mathrm{15}$ sr) at all values of centroid temperature while the V4 lidar ratios themselves are larger than the V3 value of 25 sr for all temperatures above −70 ^∘C. Based on the improved understanding of the natural variability of lidar ratios obtained from this statistical analysis, the relative uncertainty assigned to the initial lidar ratios for semi-transparent ice clouds has been reduced from 40 % in V3 to 25 % in V4.

Figure 1CALIOP ice cloud scattering parameters for both V3 and V4 plotted as functions of cloud 532 nm attenuated backscatter centroid temperature, showing (a) multiple scattering factors, (b) lidar ratios, and (c) effective lidar ratios (i.e., multiple scattering factor × lidar ratio). The multiple scattering factors and lidar ratios are sigmoid approximations to measured values reported in Garnier et al. (2015).

Download

Water clouds

For semi-transparent water clouds, there have been no changes to either the initial lidar ratio or the multiple scattering factor. In V4 these have values of 19 sr and 0.6, respectively, as they did in V3. However, the relative uncertainty in the lidar ratio has been reduced from 40 % to 15 %, reflecting substantial improvements in the cloud thermodynamic phase discrimination algorithm (Avery et al., 2018) and the implementation of a new algorithm for estimating multiple scattering factors for opaque water clouds (Hu et al., 2007).

Clouds of unknown phase

In V3, when the cloud phase could not be identified, a constant lidar ratio of 22 sr was used, this being the average of the ice cloud (25 sr) and water cloud (19 sr) values. In V4, the ice and water lidar ratios are again averaged but now the ice–cloud lidar ratio is obtained from the sigmoid approximation function shown in Fig. 1b. Similarly, the multiple scattering factors in V4 are the average of the value for semi-transparent water clouds (0.6) and the sigmoid function values shown in Fig. 1a. In V3, a constant value of 0.6 was used.

2.2.1 Increased number of constrained retrievals

Extinction profile retrievals that are constrained by measurements of the effective two-way transmittance of the layer are generally more accurate than those for which no constraint exists. Constrained retrievals generate optimized values of the layer lidar ratio that are precisely matched to the measured two-way transmittance. However, for unconstrained retrievals, optimized adjustments of the lidar ratio are not possible because no constraint is available, and hence the accuracy of the two-way transmittance estimates derived using this method depends critically on the accuracy of the initial lidar ratio. In V3, constrained retrievals were only performed when the error in the retrieved lidar ratio was estimated to be lower than 40 %. However, Garnier et al. (2015) found good agreement between CALIOP constrained optical depths and IIR optical depths at smaller values of optical depths than implied by the 40 % limit on lidar ratio uncertainty. Indeed, low optical depths correspond to high effective two-way particulate transmittances and, consequently, high lidar ratio uncertainties, as seen in Eq. (7.4) of Liu et al. (2005). By removing the lidar ratio uncertainty threshold in V4, a far greater proportion of constrained retrievals is achieved. Note, however, that the optical depth uncertainty estimates for unconstrained retrievals depend directly on the uncertainties assigned to the initial lidar ratios. In contrast, the optical depth uncertainties for constrained retrievals depend only on the uncertainties in the measured data. The uncertainties in the lidar ratios obtained from constrained solutions likewise depend on the uncertainties in the measured optical depths and are entirely independent of the uncertainties associated with the initial lidar ratios. A paradoxical consequence of this dichotomy is that for features with large uncertainties in their measured optical depths, constrained solutions can generate larger uncertainties in the retrieved backscatter and extinction profiles and derived lidar ratios than would have been the case had an unconstrained retrieval been used instead.

Constrained retrievals use measurements of the effective two-way layer transmittance and its uncertainty that are determined by comparing signals from above and below the layer. The determination of the layer boundaries is detailed in Vaughan et al. (2005, 2009). Briefly, the initial estimate of cloud base is determined as that range at which the attenuated scattering ratio (the ratio of the normalized, range-corrected backscatter signal to a molecular backscatter model) drops below a range-dependent threshold that is determined largely by signal-to-noise ratio (SNR) and signal attenuation by the overlying atmosphere. This initial estimate of cloud base is further refined by continuing to search below this altitude for a region where the attenuated scattering ratio ceases to be a decreasing function of range. Depending on the SNR in the region below the cloud, this refinement sets cloud base below any readily detectable “leakage” of the signal into the assumed clear region caused by, for example, the transient recovery time of the detectors or by multiple scattering from small particles in dense clouds. Note that this multiple scattering leakage is not pulse stretching (Miller and Stephens, 1999), but instead occurs, for instance, because the forward scattering angle from the small ice crystals in cold cirrus can be wider than the CALIOP receiver field of view (Reverdy et al., 2015), so that the multiple scattering factor can be slightly larger below the ice clouds than in cloud (Winker, 2003). Once all features have been detected in a column, so called “clear air” regions above and below each feature are identified. To initiate a constrained retrieval, the V4 CALIOP extinction algorithm requires a minimum feature-free vertical extent of 2.48 km both above and below a candidate feature. The required effective layer two-way transmittance is then calculated as the ratio of the mean attenuated scattering ratios computed over these below-cloud and above-cloud clear air regions. The fidelity of these estimates relies on the supposition that the backscatter signals in the clear air regions are due solely to air molecules. If this condition is not met (and this is impossible to confirm with absolute certainty), then unless the mean particulate scattering ratios in the two clear air regions are identical, the transmittance measurements will be in error, no matter how small the reported uncertainty, and the constrained retrieval will also be in error. These are bias errors, not random errors, as discussed in Sect. 3b2 of Young et al. (2013) and by del Guasta (1998). Undetected particulate layers above the layer being analyzed can also affect the calculated lidar ratio. Extreme errors can cause the derived lidar ratios to approach and sometimes even exceed the physically acceptable limits of 0.05 to 250 sr imposed by the V4 retrieval scheme. Any constrained retrieval (bit 0 set to 1 in the extinction QC flag – See Sect. 2.2.6) in which the derived lidar ratio is equal to either of these limits is to be treated as suspect. When serious errors are encountered in the solution of constrained retrievals, bit 8 is also set, giving an extinction QC flag value of 257.

2.2.2 Criteria for determining successful retrievals

The criteria for determining whether a retrieval is successful or not have been modified in V4. The equation for calculating the particulate backscatter at a given range is transcendental and is solved iteratively using a Newton–Raphson algorithm (YV09), while the equation for the corresponding uncertainty is now a simple algebraic expression. In V3 it was solved iteratively. (An improved initialization of the backscatter solution used in V4 is provided in Sect. S2 in the Supplement) As shown in Sect. S1, the lidar ratio appears in the equations for both the particulate backscatter and its uncertainty at a given range. It is simple to determine if a given lidar ratio is too large for the current input data, because solutions simply do not exist. In V3, retrieved uncertainties in backscatter, extinction, and optical depth that grew too large were assigned a fixed value of 99.99. However, in V4, the non-existence of an uncertainty solution or the retrieval of an uncertainty that is excessively large are both considered, along with the non-existence of a backscatter solution, to be indications that the lidar ratio is too large. These conditions trigger an algorithm that reduces the lidar ratio. Having the number of iterations to achieve convergence of the backscatter solution exceed some prescribed maximum is also an indication that the lidar ratio is too large.

2.2.3 New algorithm for retrievals in opaque layers

The sensitivity of extinction and optical depth retrievals to lidar ratio errors increases dramatically as the layer optical depth increases (Y1316). In layers determined to be opaque, even small lidar ratio errors can cause large errors in the retrievals. In V3 and previous versions, the first retrieval attempts were made using the initial lidar ratio prescribed for the layer type and subtype. As it is obviously impossible to select an initial lidar ratio that exactly matches the true value in every actual layer studied; the initial value is virtually certain to be either too large or too small. The former case is easily identified, as it is not possible to retrieve a complete profile if the lidar ratio is too large. In these situations, the earlier algorithms reduced the lidar ratio, initially in steps of 1 %, and then in steps of 5 %, until a successful retrieval was obtained (YV09). However, given the extreme sensitivity of the retrieval to the lidar ratio and the relatively coarse reductions, the final values of these reduced lidar ratios may well have been too small. The latter case is more difficult to identify, as successful (although underestimated) retrievals using the initial lidar ratio (QC =16) can still be obtained even if the initial value is too small.

In order to assess the incidence and likely impact of V3 lidar ratios that were either too high or too low, we examined all opaque ROI clouds detected in the V3 data set during January 2010, with separate analyses done for daytime and nighttime measurements. To estimate the lidar ratio that would be required for a totally attenuating layer, we use a well-validated method initially developed by Platt (1973). Given values for the integrated attenuated particulate backscatter profile through the layer, ${\mathit{\gamma }}_P^{\prime }$, the layer effective two-way transmittance, $T_P^{\mathrm{2}}$, and the layer multiple scattering factor, η, the lidar ratio can be estimated using

The layer effective two-way transmittance, $T_P^{\mathrm{2}}$, is zero for totally attenuating layers, so that Eq. (1) simplifies to $S=\mathrm{1}/\left(\mathrm{2}\mathit{\eta }{\mathit{\gamma }}_P^{\prime }\right)$. For the V3 analyses, η for ice clouds is fixed at 0.6, and ${\mathit{\gamma }}_P^{\prime }$ is obtained from the V3 data products. These ${\mathit{\gamma }}_P^{\prime }$ values are corrected for molecular scattering contributions using the steps described in Sect. 3.2.9.1 of Vaughan et al. (2005). To eliminate artifacts due to failed surface detections in V3, we require ${\mathit{\gamma }}_P^{\prime }$ to be greater than 0.0104 sr⁻¹, equivalent to an opaque layer lidar ratio of 80 sr. Enforcing this restriction eliminated 0.4 % of the initial data set.

For those V3 retrievals where the lidar ratio was not reduced (i.e., QC =16), the final lidar ratio is, as expected, identical to the initial lidar ratio at 25 sr. However, the lidar ratio required to attenuate the signal totally (i.e., for $T_P^{\mathrm{2}}=\mathrm{0}$), computed using Eq. (1), is larger by ∼24 % at night ($S_{\mathrm{night}}=\mathrm{30.9}\pm \mathrm{4.4}$ sr for 79 049 samples) and by ∼32 % for daytime measurements ($S_{\mathrm{day}}=\mathrm{32.9}\pm \mathrm{6.0}$ sr for 82 101 samples). On those occasions when the lidar ratio was reduced (QC =18; 19 900 nighttime samples and 21 098 daytime samples), the mean final lidar ratios in the V3 data products were 21.1±3.0 sr at night and 20.8±3.3 during the day. For these same cases, the mean values of the “totally attenuating” lidar ratios computed using Eq. (1) are larger by ∼3 % at night ($S_{\mathrm{night}}=\mathrm{21.7}\pm \mathrm{3.0}$ sr) and ∼5 % during the day ($S_{\mathrm{day}}=\mathrm{21.8}\pm \mathrm{3.6}$ sr). The lower lidar ratios reported in the V3 data products for the QC =18 solutions are most likely caused by the larger step sizes used to reduce the lidar ratio estimate in the V3 extinction retrieval. While these QC =18 lidar ratio differences are small, the errors introduced in subsequent calculations can be quite large. For example, following the error sensitivity analyses in Y1316, and assuming an optical depth of 3.5, an underestimate of 3 % in the lidar ratio is expected to produce a retrieved optical depth that is, on average, 50 % too low.

From this study we draw two conclusions. First, the initial lidar ratio used for opaque ice clouds in the V3 and earlier data products was, in general, far too low. As a direct consequence, the retrieved extinction coefficients and the estimate of optical penetration into the clouds (i.e., the effective optical depth) were also significantly underestimated. Second, the magnitudes of the lidar ratio reductions were typically too coarse, resulting in an underestimate of lidar ratio even in those cases where the lidar ratio was reduced. Both of these deficiencies have been addressed in the V4 algorithm. The initial value of the lidar ratio for opaque layers is now obtained using Eq. (1), with the effective two-way transmittance assumed to be zero. The multiple scattering factor is derived according to the type of layer being analyzed. For ice clouds, an initial temperature-dependent multiple scattering factor is computed based on the same attenuated backscatter centroid temperature as for semi-transparent clouds (Garnier et al., 2015). For water clouds, the multiple scattering factor is derived from the layer-integrated volume depolarization ratio using the method of Hu (2007). For aerosols, multiple scattering is assumed to be negligible, and hence η is fixed at 1.0. However, this assumption may not be valid for opaque aerosol layers, and must be considered especially tenuous for opaque dust layers (Wandinger et al., 2010; Liu et al., 2011). In these cases, the η=1 assumption introduces bias errors similar to those incurred by the incorrect specification of lidar ratio (e.g., Y1316). Fortunately, because opaque aerosol layers occur only infrequently (∼1 % of all unique aerosol layers, amounting to ∼0.2 % of all unique layers, detected in 2012 were identified as being opaque), the effects of these bias errors are somewhat mitigated. To increase the accuracy of the initial lidar ratios computed using Eq. (1), these values are further refined using an iterated version of Eq. (15) in Fernald et al. (1972). (Mathematical details are provided in Sect. S3). Additionally, V4 reductions in the lidar ratio (see Sect. 2.2.4) no longer used fixed, predefined increments, but instead are made inversely proportional to the average retrieved extinction in the layer, so as to make finer adjustments in those cases where the retrieval is more sensitive to lidar ratio errors.

For ice clouds, determining the lidar ratio is a two-step process. In the first step, the multiple scattering factor and an initial lidar ratio are calculated based on the temperature at the centroid of the attenuated backscatter profile (Sect. 2.1.2 – Ice clouds). This lidar ratio is recorded in the V4 data products as the initial lidar ratio. After a successful retrieval of the particulate backscatter and extinction over the whole detected depth of the layer, the multiple scattering factor is recalculated at the altitude of the retrieved particulate backscatter centroid. This recalculation is not necessary for semi-transparent layers, where the centroid altitudes of the attenuated backscatter profiles provide reliable approximations of the centroid altitudes of the particulate backscatter profiles. However, for opaque layers, the differences between the two can be large (up to a kilometer or more) and hence the multiple scattering factors at the two altitudes and temperatures can be quite different. As the recalculated centroid height will be at a lower altitude (and likely a warmer temperature) than that of the original attenuated backscatter centroid, the updated multiple scattering factor is likely to be lower than the initial estimate (see Fig. 1a). An updated extinction solution is then calculated using the updated multiple scattering factor and the newly derived estimate of lidar ratio. The optimized lidar ratio obtained from this second solution is recorded in the data products as the final lidar ratio. One consequence of the refinements made to the multiple scattering factor in this two-step solution process is that the final lidar ratio reported in the V4 data products for opaque layers can sometimes be larger than the initial lidar ratio.

It should be noted that the V4 surface detection algorithm (Vaughan et al., 2018b) has led to large improvements in the assessment of what is and what is not an opaque layer and, as a consequence, when the opaque-layer algorithm can be used. Nevertheless, the accuracy of the new algorithm can still sometimes be compromised by the low SNR conditions that can occur, for example, during daytime measurements of bright, opaque clouds. Low SNR may lead both to missed detections by the surface detection scheme and, within the layer detection algorithm, to missed detections of weaker backscatter signals near the top and apparent base of the opaque layer. The former problem causes the layer two-way transmittance to be set incorrectly to zero and the latter causes the calculated layer integrated attenuated backscatter to be too small. Both of these errors lead to an overestimate of the initial lidar ratio. In such cases the retrieved extinction profile and optical depth will likewise both be overestimated.

In Fig. 2, we use simulated data to demonstrate the effect of the different V3 and V4 algorithms on the retrieved extinction profiles. The top of the simulated cloud layer is at 10 km while the base is at 4 km, although it can be seen in Fig. 2a that the signal is totally attenuated at an altitude of about 7.5 km. The V3 retrieval (Fig. 2b), using a fixed lidar ratio of 25 sr and multiple scattering factor of 0.6, can be seen to trend quickly to zero, significantly underestimating the extinction coefficients at all altitudes. In contrast, the V4 retrieval, which obtains its lidar ratio from the data, stays much closer to the target, trending slightly too low from cloud top to ∼8.8 km, then somewhat too high to around 8 km, after which the solution oscillates between being too high and too low. These excursions from the target result from a combination of small differences in the estimated lidar ratio from the target value and from the effects of noise on the signal.

Figure 2A comparison of the extinction profiles retrieved by the V4 and V3 algorithms in a simulated totally attenuating ice cloud. The simulated layer has its top at 10 km and its base at 4 km, with a constant extinction coefficient of 2 km⁻¹, yielding an optical depth of 12. The cloud lidar ratio and multiple scattering factor are also uniformly constant values of 33.5 sr and 0.52, respectively. Panel (a) shows a height vs. latitude image of the simulated 532 nm attenuated backscatter coefficients. Panel (b) shows the mean extinction coefficients, averaged across the full horizontal extent of the layer, retrieved by the V3 scheme and the V4 scheme. For both versions, individual extinction profiles were retrieved at 5 km intervals prior to averaging.

Download

2.2.4 Revised algorithms for adjusting lidar ratios to achieve a successful solution

There are four main potential errors in the inputs supplied to the extinction algorithm. Two of these errors cause the retrieved particulate extinction to be too high, or cause there to be no possible solution at all. The other two cause the extinction to be too low, or even negative.

If the initial lidar ratio estimate is too high, or the correction for the attenuation caused by overlying layers is too large (i.e., when the optical depths of overlying layers are overestimated), then the extinction retrieved in the lower layers will be too high, or, in the worst case, not retrievable at all. While retrieval problems in the topmost layers in the atmosphere can usually be attributed to errors in the initial lidar ratios for these layers, as the retrieval process progresses to lower layers that have an increasing number of overlying layers, the retrieval problems are increasingly likely to be at least partially the result of attenuation correction errors. As there is no completely reliable way of determining which of these error sources causes an unsuccessful retrieval, unsuccessful retrievals are addressed by reducing the lidar ratio in both cases. Sometimes the lidar ratio needs to be reduced to values that are far below what is considered to be a valid range. These cases are considered to be failed retrievals and are terminated with the QC flag being set to indicate the problem. Where V4 retrievals are terminated at an altitude above the detected base of the feature, a fill value of −333 is inserted in the retrieval arrays of particulate backscatter and extinction, and their uncertainties, for this altitude and all lower altitudes down to the detected layer base.

When a successful retrieval could not be achieved using a given lidar ratio, the V3 extinction algorithm reduced the lidar ratio by repeatedly applying fixed fractional reductions, initially in steps of 1 % then in steps of 5 %, until either a complete solution was derived or some error condition (e.g., too many solution attempts) occurred [YV09]. The V4 algorithm takes a more nuanced approach. V4 adjustments to the lidar ratio in opaque layers are inversely proportional to the average extinction coefficient retrieved over the path from the top of the layer to the point where the retrieval fails with the current lidar ratio, multiplied by both the retrieved two-way particulate transmittance over the same range and an empirically derived scaling constant. This produces smaller adjustments where the extinction is higher, as is required to avoid overcorrections. Lidar ratio reductions in opaque layers are limited to a maximum of 1 % on each adjustment. For semi-transparent layers, the fractional reduction is 10 % of the relative uncertainty in the initial lidar ratio.

The opposite problem of the retrieved extinction being too small (or even negative) can occur if the initial lidar ratio is too small or the estimated attenuation for overlying layers is too low. As shown in Y1316, when the optical depth of the layer being analyzed is high, these underestimates can introduce significant errors in the extinction retrieval (e.g., as shown in Fig. 2b). However, whereas cases for which the lidar ratio or the overhead attenuation correction is too large are readily identified because the extinction solution will diverge before reaching the bottom of a layer, we know of no reliable metric that can unambiguously identify those instances where the lidar ratio or the overhead attenuation correction is too low. Consequently, unlike the V3 algorithm, the V4 retrieval does not increase the lidar ratio when negative particulate backscatter coefficients are repeatedly retrieved from positive attenuated backscatter inputs.

As in V3, retrieval uncertainties and retrieval errors in overlying layers propagate into and are amplified by extinction retrievals in underlying layers. While the uncertainties reported in the V4 data products correctly account for the effects of estimated random errors in overlying layers, no estimates are provided for possible bias errors that might also be incurred. However, the combined random uncertainty in the optical depths of overlying layers can be used to estimate the bias error introduced by the accumulated attenuation corrections. Estimates of the total bias error in the lower layer can then be obtained using the various equations and figures in Y1316.

2.2.5 Modification of lidar ratio uncertainties

Lidar ratio uncertainties are major contributors to the extinction and optical depth uncertainties reported in the data products (Y1316). The initial lidar ratio uncertainties are assigned by the various lidar ratio determination schemes (i.e., as in Table 1). However, there are some circumstances where the initial lidar ratio uncertainty is modified, or even recalculated completely, within the V4 extinction retrieval algorithm. This final value, whether modified or not, is reported in the V4 data products as the final lidar ratio uncertainty and is expressed as an absolute uncertainty in steradians.

Lidar ratio uncertainties are calculated in constrained retrievals from the uncertainties in the measured apparent two-way particulate transmittance of the layer under analysis and the integrated particulate attenuated backscatter of the layer, as in Vaughan et al. (2005). In V4, this uncertainty may be reduced if the transmittance plus or minus its uncertainty exceeds the range [0, 1] or if the calculated lidar ratio plus or minus its uncertainty exceeds the range of acceptable values (currently 0.05 to 250 sr). In opaque features, the lidar ratio uncertainty may also be reduced from the default value using the range of lidar ratios calculated from estimates of the minimum and maximum two-way transmittance of the layer, where the minimum is zero (total attenuation) and the maximum is that value that would produce a particulate backscatter at the detected layer base that is greater than zero where the attenuated backscatter is also greater than zero. If these recalculations produce values that are out of the acceptable range, then the relative uncertainty in the default value is retained. When lidar ratios need to be reduced in order to retrieve a complete profile over the whole depth of the layer, the reported final lidar ratio uncertainty is likewise reduced so as to maintain the relative uncertainty that is calculated either from the default values or from the modified values just described.

The rules described above do not apply for lidar ratio uncertainties for opaque water clouds because of lidar “pulse stretching”. Pulse stretching is caused by the broad forward peak of the scattering phase function of the small spherical droplets typically found in water clouds. The broad forward peak causes an increase in off-axis scattering in water clouds. With the increased multiple scattering found at high optical depths (e.g., in opaque water clouds), this off-axis scattering quickly dominates the backscattered signal with increasing penetration into the layer. As lidar range information is measured by the time delay of the backscattered signal, off-axis scattering causes errors in the measured delay, and thereby causes errors in cloud penetration depth measurements. Owing to this loss of range information, the particulate backscatter and extinction uncertainties reported for opaque water clouds in the CALIOP V4 data products are now assigned a fill value of −29, indicating zero confidence in the relationship of the retrieved variables to the related altitude array. However, we choose to continue to report the retrieved backscatter and extinction for data users who may find value in these quantities as a function of time.

2.2.6 Changes in the extinction quality control (QC) flags

In order to accommodate the modifications described above, and other changes to the algorithms, the extinction QC flags have been modified in V4. The new values and their explanations are shown in Table 2. The relative occurrence of each of the QC bits for both clouds and aerosols is provided in Table 3. It can be seen that while nearly 93 % of the aerosol layers have identical QC flags in V3 and V4, identical QC values are found in only 65 % of cloud layers. This difference results mainly from the much higher occurrence of QC =0 in retrievals in aerosol layers.

Table 2Extinction quality control flags used in version 3 and version 4.

Download Print Version | Download XLSX

Table 3A comparison of the V4 and V3 extinction QC flag values for identical layers detected during the years 2007–2010. All QC flag values not shown separately are grouped under QC =99 and indicate the proportion of unsuccessful retrievals: (a) clouds, (b) aerosols.

Download Print Version | Download XLSX

3.2.1 Ice clouds

The new opaque layer analysis permits the retrieval of the particulate lidar ratio, a parameter of considerable interest. The accuracy of the lidar ratio depends on the accuracy of the integrated attenuated particulate backscatter of the layer and of the determination of its opacity. The accuracy of the first depends on accurate calibration, accurate determination of the existence of overlying layers and correction for their attenuation, and accurate determination of the full vertical extent of the layer down to the altitude at which the backscatter signal is extinguished (i.e., where the particulate transmittance is truly zero). The accuracy of the second depends on the reliable determination of complete opacity (particulate transmittance is zero) by reliably determining the absence of any surface backscatter signals below the layer. All of these factors are affected by the signal-to-noise ratio, which is lower during the day, so nighttime measurements are expected to be to be more reliable. As an example, we present an analysis of the topmost cloud layers measured during the period 2012–2015 and composed of ROI particles. As seen in Fig. 6a, the geometric vertical extent is clearly smaller for daytime data (median $=\mathrm{2.6}\pm \mathrm{1.1}$ km) than for nighttime data (3.7±1.6 km). The distributions of the integrated attenuated backscatter shown in Fig. 6b are wider during daytime than during nighttime and are slightly skewed towards smaller values. Nevertheless, because the extra geometric extent of the nighttime clouds is at the base, where the signal is highly attenuated, the day and night median integrated attenuated backscatters differ by only 3.3 %. In V3, the initial lidar ratio was 25 sr and the multiple scattering factor was 0.6, which (using Eq. 1) corresponds to a layer integrated attenuated backscatter of 0.033 sr⁻¹. The median measured values of the V4 integrated attenuated backscatters differ from this value by ∼6 % (daytime) to ∼10 % (nighttime). However, as seen in Fig. 6c, the V4 lidar ratios retrieved from these opaque layers are considerably larger than the V3 initial lidar ratio, at 31.5±8.1 sr during the daytime and 31.8±6.7 sr at night. (Recall that the final lidar ratios in V3 were never larger than the initial lidar ratio of 25 sr.)

Figure 6Histograms of (a) cloud geometric extent, (b) 532 nm integrated attenuated backscatter, and (c) 532 nm lidar ratio for V4 opaque clouds composed of randomly oriented ice (ROI) particles measured during the period 2012–2015. Data are for the top layer in the column only. Statistics are median ± median absolute deviation.

Download

To demonstrate that the new algorithm is indeed producing lidar ratios that are closer to what is expected in opaque clouds than found for V3 retrievals in Sect. 2.2.3, we repeated our analysis of opaque ROI clouds detected in January 2010 using the V4 algorithm. For retrievals in which the lidar ratio was not reduced (QC =16), we found that the final lidar ratio was within 8 % of the value expected for total attenuation (as calculated using Eq. 1) for daytime retrievals and only 1.5 % for nighttime retrievals. This is a very marked improvement over the V3 results where the corresponding values were 32 % and 24 %, respectively. For retrievals in which the lidar ratio needed to be reduced in order to achieve a solution (QC =18), the final V4 lidar ratios were within 4.8 % of the value expected for total attenuation for daytime retrievals and 1.8 % for the nighttime retrievals. The corresponding V3 values were 4.4 % and 2.9 %. It can be seen that the major improvement is achieved in the QC =16 retrievals where the V3 lidar ratios were markedly too low and would have produced retrieved extinction profiles that were also too low (Fig. 2). This improvement results from the better selection of the initial lidar ratio in V4. A less marked improvement has been achieved in the QC =18 retrievals. This is because the initial V3 lidar ratio was too high, rather than too low, and was subsequently reduced sufficiently to achieve a solution. The marked improvement in V4 is found in the nighttime retrievals. In the January 2010 sample analyzed, QC =18 retrievals occurred more than twice as often in the analysis of nighttime data than in the daytime data. This improvement results mainly from the more finely tuned lidar ratio reduction scheme employed in V4.

In Fig. 7 we present maps of the global variations in 532 nm lidar ratios for opaque ROI clouds measured during daytime (Fig. 7a) and nighttime (Fig. 7b) during the years 2012 to 2015. The data are for the same clouds as used to create the histograms in Fig. 6. Given the large differences in daytime and nighttime calibrations, the excellent agreement shown between the distributions of daytime and nighttime lidar ratios is remarkable and gives confidence both in the calibrations and in the new opaque-layer algorithm. The agreement also suggests that the lidar ratio and, by implication, the microphysics in the uppermost regions of opaque ice clouds, do not change significantly between day and night.

A considerable latitudinal variation can be seen in the lidar ratios with the lowest values found over the tropical region and over Antarctica. If the temperature dependence of the lidar ratio for opaque ROI clouds is the same as that for semi-transparent clouds, this behavior could, at least in part, be explained by the variation in lidar ratio with temperature shown in Fig. 1b. Indeed, in the tropics and over Antarctica, the ice clouds occur at altitudes where the temperature is the coolest, as shown in Fig. 7c and d. There are also some land–ocean differences, which suggest that there could be geographic influences on the cloud microphysics. A more detailed analysis, which is beyond the scope of this present work, would be required to determine the causes of the observed distribution.

Figure 7Global distribution of lidar ratios measured in opaque ROI clouds (a) by day and (b) by night between 2012 and 2015. The corresponding cloud 532 nm attenuated backscatter centroid temperatures are shown in panels (c) and (d).

Download

3.2.2 Water clouds

The new opaque layer algorithms described in Sect. 2.2.3 have also been used to determine the lidar ratio for water clouds. Although retrieval of extinction profiles in opaque water clouds is of low-to-no confidence (as indicated by an uncertainty fill value of −29), the lidar ratio determined from these clouds gives an indication of the possible variation of the lidar ratios that should be used in retrievals in semi-transparent water clouds. In Table 4, we present globally averaged data for the period 2008 through 2010 on the lidar ratio retrieved for opaque water clouds measured during the daytime and nighttime, and on the parameters used in its calculation. Filters have been applied to the data to minimize the effects of undetected overlying layers, to ensure that the layers are composed only of liquid water droplets and that the layers are sufficiently robust to ensure reliable results.

Table 4Statistical characterization of global measurements of opaque water clouds during 2008–2010, where the clouds were the only layers detected in a 5 km averaged column. The integrated attenuated backscatter, layer-integrated volume depolarization ratio, and layer-integrated attenuated backscatter color ratio are measured over the detected cloud geometrical thicknesses. The multiple scattering factors are derived using the method of Hu et al. (2007), and the lidar ratios are retrieved using Eq. (1). MAD is the median absolute deviation and N is the number of unique cloud layers used in the calculations. Data have been filtered to minimize the effects of potentially undetected overlying layers. Layer integrated attenuated backscatter is limited to the range (0.021–0.111 sr⁻¹), layer depolarization to the range (0.03–0.39), layer attenuated color ratio to the range (0.90–1.50) and overlying attenuated backscatter to values less than 0.02 sr⁻¹, and (for nighttime layers only) greater than zero.

Download Print Version | Download XLSX

Figure 8Mean global distribution of daytime opaque water clouds during 2008–2010; (a) 532 nm lidar ratios, (b) sample counts. Data have been filtered as described in the caption for Table 4.

Download

We note that the mean daytime lidar ratio of 18.5±2.8 sr is very close to the theoretically expected value, while the nighttime mean is somewhat lower at 17.2±2.9 sr. The low bias in the nighttime lidar ratios is thought to result from a combination of the slow transient recovery of the 532 nm detectors in response to the extremely strong signals from these dense water clouds (Hunt et al., 2009) and a diurnal variation in the instrument's polarization gain ratio (PGR) that is used to calculate the depolarization ratios. The non-ideal detector response is responsible for the larger magnitudes of the nighttime integrated attenuated backscatter. The diurnally varying PGR yields depolarization ratios are slightly, but persistently, lower at night leading to multiple scattering factors that are slightly, but persistently, higher. This combination of high biases in integrated attenuated backscatters and multiple scattering factors at night leads to low biases in the nighttime lidar ratio estimates.

The mean global distribution of the lidar ratios measured in daytime opaque water clouds during 2008–2010 is shown in Fig. 8. It is immediately apparent that this distribution is quite different from that of the opaque ice clouds shown in Fig. 7. Generally, higher lidar ratios are seen in the Northern Hemisphere compared with these in the Southern Hemisphere. Higher lidar ratios are seen over the land, and adjacent oceanic regions, particularly in the regions influenced by dust in North Africa and the Middle East, Central Asia and China. Regions influenced by smoke in South America and southern Africa also have higher lidar ratios. While it is tempting to explain the higher lidar ratios in these regions in terms of the Twomey effect (Twomey, 1977), where greater numbers of condensation nuclei produce greater numbers of smaller cloud droplets, or by the increase in extinction through the inclusion of absorbing particles within the cloud water droplets (Mishchenko et al., 2014; Chylek and Hallett, 1992; Miles et al., 2000; Wittbom et al., 2014), a more thorough analysis is required before we can reach these conclusions. Apparent increases in lidar ratio can also be caused by SNR limitations, (particularly in the daytime), which appear as undetected overlying aerosol layers, incomplete determination of the full extent of the cloud layer, or incorrect assignment of opacity because of the missed detection of underlying layers or the surface. While a more thorough analysis is beyond the scope of this current paper, the results presented here suggest that further research could be rewarding.

3.3.1 Opaque clouds

We turn now to the fractional optical depths retrieved in opaque clouds using the new methods. Readers are reminded that the CALIOP optical depths reported for opaque clouds are retrieved only over the range from the top of the cloud down to the altitude at which the signal is totally attenuated (i.e., when the signal is indistinguishable from the ambient noise). Consequently, CALIOP opaque cloud optical depth estimates cannot be compared with cloud optical depths measured by passive instruments. In Fig. 9, we present a comparison of the V3 and V4 distributions of optical depths for the topmost opaque (QC =16 and QC =18) ROI clouds detected during 2012–2015. There are several features of note. Most obvious is the marked difference between the V3 and V4 mean optical depths for these opaque clouds. The V3 distributions for day and night peak at optical depths in the range 1.2 to 1.5, or effective optical depths of 0.72 to 0.9. Such low values are insufficient to cause complete attenuation of the CALIOP signal and reinforce our conclusion that the V3 lidar ratios, and, consequently, optical depths, are considerably underestimated.

Figure 9Distributions of the apparent optical depths measured in opaque ROI clouds during 2012–2015 and processed using the V3 and V4 algorithms.

Download

The second notable feature is the large difference in the daytime and nighttime V4 distributions. This difference is explained by the dramatic shift from QC =16 to QC =18 that occurred in the V4 daytime opaque cloud retrievals. The main nighttime peak at ∼6.6 corresponds to retrievals with QC =16. Optical depths smaller than about 6 correspond to the far fewer nighttime retrievals in which the lidar ratio required reduction (QC =18). Conversely, for the daytime data, the small secondary peak at an optical depth of ∼6.6 corresponds to relatively few retrievals in which the lidar ratio has not been reduced (QC =16), whereas the much larger peak at ∼4 corresponds to those retrievals where the lidar ratio has been reduced. The much broader daytime QC =18 peak in V4, when compared with the sharper peaks in V3, results largely from the new lidar ratio reduction scheme with its finer and variable adjustments.

The final feature of note relates to the peaks in the frequency of occurrence of V3 optical depths at around 2.5 and 3.85. When multiplied by a multiple scattering factor of 0.6 that was used for clouds in V3, these values correspond to effective optical depths of 1.5 and 2.5, respectively. The peaks occur because of the fixed reduction steps applied in the V3 algorithms to the lidar ratio, in the case of a solution that would not converge. The peak at an optical depth of 2.5 corresponds to a lidar ratio reduction of 5 %, and that at 3.85 to a reduction of 1 %. These reductions are too coarse in optically dense layers, where the transmittance is close to zero, and cause the artifacts seen in the figure. (See Omar et al., 2009 for a detailed explanation.) These artifacts are not seen in V4 thanks to the refined reduction of the lidar ratios described in Sect. 2.2.4.

Figure 10Median cloud optical depths (CODs) for semi-transparent, high-confidence nighttime ROI clouds for the period March to May 2010 inclusive, nighttime only. The area-weighted average optical depths from the maps are 0.137 for V3 (77 230 457 samples) and 0.266 for V4 (78 808 302 samples).

Download

3.3.2 Semi-transparent clouds

Global maps of the nighttime optical depths for semi-transparent high confidence ROI clouds (QC =0, 1 or 2) are presented in Fig. 10. For both V3 (Fig. 10a) and V4 (Fig. 10b), the optical depths in the tropical regions, particularly over the oceans, are considerably lower than at higher latitudes. The V4/V3 COD ratios (Fig. 10c) are higher at middle to high latitudes and suggest that the changes between V4 and V3 differ, to some extent, over land and ocean. For example, the lower V4/V3 ratios over South America, Central and Southern Africa, and especially over the Himalayas are rather distinct. In general, the area-weighted average optical depths are considerably higher for V4 (0.266) than for V3 (0.137). The dominant cause of the larger V4 optical depths is the higher lidar ratios (Fig. 1b) used in the V4 retrievals.

As mentioned previously, Holz et al. (2016) showed that while CALIOP V3 constrained (QC =1) optical depths in semi-transparent ice clouds agreed rather well with MODIS C6 optical depths, the unconstrained retrievals (QC =0, 2) were substantially smaller than the MODIS values. As a conclusive demonstration of the major improvement in the V4 optical depths, in Fig. 11 we compare MODIS C6 optical depth estimates for semi-transparent ice clouds measured over the ocean during daytime, with collocated CALIOP V3 and V4 optical depths. The MODIS 1 km C6 optical depths were collocated with the centers of the CALIOP 5 km data segments. The samples selected for comparison were limited to those cases for which only one high-confidence ROI cloud was detected in the CALIOP 5 km column and no clouds were detected at single shot resolution and subsequently removed by the boundary layer cloud clearing algorithm. The layers selected for comparison were required to have identical base and top altitudes in both V3 and V4, and they were further restricted to those cases for which the extinction QC flag was either 0 or 1. Data for January 2008 are shown in the upper row in Fig. 11 while data for July 2008 are shown in the lower row. Whereas the CALIOP V3 data presented in Fig. 11a and c are markedly lower that the MODIS C6 values, the comparisons in Fig. 11b and c show an excellent agreement of the V4 optical depths with the MODIS values. The good agreement of the CALIOP V4 and MODIS C6 optical depths gives us confidence in the validity of the new multiple scattering factors and lidar ratios presented in Sect. 2.1.2.