The Cloud–Aerosol Lidar with Orthogonal Polarization (CALIOP) on board the Cloud–Aerosol Lidar Infrared Pathfinder Satellite Observations (CALIPSO) satellite has been making near-global height-resolved measurements of cloud and aerosol layers since mid-June 2006. Version 4.10 (V4) of the CALIOP data products, released in November 2016, introduces extensive upgrades to the algorithms used to retrieve the spatial and optical properties of these layers, and thus there are both obvious and subtle differences between V4 and previous data releases. This paper describes the improvements made to the extinction retrieval algorithms and illustrates the impacts of these changes on the extinction and optical depth estimates reported in the CALIPSO lidar level 2 data products. The lidar ratios for both aerosols and ice clouds are generally higher than in previous data releases, resulting in generally higher extinction coefficients and optical depths in V4. A newly implemented algorithm for retrieving extinction coefficients in opaque layers is described and its impact examined. Precise lidar ratio estimates are also retrieved in these opaque layers. For semi-transparent cirrus clouds, comparisons between CALIOP V4 optical depths and the optical depths reported by MODIS collection 6 show substantial improvements relative to earlier comparisons between CALIOP version 3 and MODIS collection 5.
The Cloud–Aerosol Lidar with Orthogonal Polarization (CALIOP) lidar (Hunt et
al., 2009) on board the Cloud–Aerosol Lidar Infrared Pathfinder Satellite
Observations (CALIPSO) satellite (Winker et al., 2010) has now acquired over
12 years of global, height-resolved information on the location, extent, and
optical properties of clouds and aerosols between the latitudes of 82
The tasks accomplished by the extinction algorithms include the retrieval of all of the particulate optical profiles, including backscatter coefficients, extinction coefficients and their layer integrals (i.e., optical depths), particulate depolarization ratios, and particulate color ratios (the ratio of the particulate backscatter coefficients at 1064 and 532 nm). The extinction algorithm requires initial estimates of the particulate extinction-to-backscatter ratios, called lidar ratios, to retrieve extinction and optical depth from the measured attenuated backscatter. These lidar ratios vary depending on the type of particle (ice cloud, water cloud, or aerosol subtype), as determined by the preceding algorithms. Whenever relevant, contributions from multiple scattering are taken into account through a multiple scattering factor (YV09). The initially assigned lidar ratios and multiple scattering factors, called analysis parameters, are often critically important drivers of the retrieved extinction and optical depth. A notable exception is when the retrievals can be constrained by estimates of the feature two-way transmittance, which can be directly derived from CALIOP attenuated backscatter measurements made immediately above and below the boundaries of semi-transparent features (YV09). This procedure, already implemented in version 3 (V3), has been extended in V4. In the case of opaque layers, the initial lidar ratio in V4 is estimated from measurements rather than being assigned based on layer type, as was done in V3 (Sect. 2.2.3). This important change is paired with a totally new dedicated surface detection algorithm (Vaughan et al., 2018b), which leads to a significantly more reliable identification of opaque layers than in V3. When the retrievals cannot be constrained or when the initial lidar ratio cannot be determined directly from the measurements, the extinction algorithm uses a set of initial lidar ratios, many of which have been significantly changed in V4.
In addition to changes to the extinction retrieval algorithms and the analysis parameters, significant upgrades to preceding algorithms in the analysis chain also affect the retrieved quantities. For example, the new 532 nm nighttime calibration algorithm (Kar et al., 2018) normalizes the measured attenuated backscatter profiles to a molecular model at higher altitudes than in the previous versions, in order to avoid errors caused by the presence of aerosols in the supposedly aerosol-free calibration region. This can lead to significant changes to the attenuated backscatter, particularly in the tropics, and would be expected, generally, to cause increases in the retrieved optical quantities. The number and vertical and horizontal extent of the features subsequently detected have been impacted by changes in the calibration procedures (Kar et al., 2018; Getzewich et al., 2018) and the use of an improved molecular model (Kar et al., 2018). The retrieval of optical property profiles has also been affected by upgrades to the cloud–aerosol discrimination (CAD) algorithm (Liu et al., 2018). Large changes in calibration, especially at 1064 nm (Vaughan et al., 2018a), necessitated a complete restructuring of the probability density functions that drive the CAD algorithm, and hence the classification of features as clouds or aerosols is not uniformly consistent from V3 to V4. Additionally, the generic stratospheric layer type identified in V3 has been eliminated in V4, and the CAD algorithm is now applied to all features detected in both the troposphere and the stratosphere. The algorithm that classifies clouds as water or ice has been refined in V4 (Avery et al., 2018). Similarly, the aerosol subtyping algorithms have been extensively revised, and the characteristic lidar ratios for several aerosol subtypes have been updated to reflect ongoing research carried out over the past decade. V4 now includes a new tropospheric aerosol subtype (dusty marine) and several stratospheric aerosol subtypes (Kim et al., 2018).
Although changes to the algorithms that precede the extinction retrieval algorithm can be expected to impact the extinction retrieval, this paper concentrates on the description of changes to the extinction algorithm and its associated analysis parameters. Details of the extinction and optical depth changes attributed to the upstream algorithms will be published separately (e.g., Kim et al., 2018).
The outline of this paper is as follows. Section 2 describes changes to the extinction retrieval process, beginning with a description of changes to the analysis parameters (the lidar ratios and multiple scattering factors for clouds and aerosols), then proceeds to a description of changes to the various components of the actual extinction retrieval algorithm, which includes the description of a new algorithm for retrievals in opaque layers. Section 2 also describes revised algorithms for adjusting lidar ratios to achieve successful solutions, modification of lidar ratio uncertainties, and changes to the extinction quality control (QC) flags, which report the status of the extinction solution at the conclusion of the retrieval process. Section 3 presents the results of these changes and includes comparisons of the V4 and V3 retrievals in tropical clouds, lidar ratios retrieved in ice and water clouds using the new opaque layer algorithm, optical depths in opaque and semi-transparent clouds, and median profiles of extinction coefficient as a function of mid-cloud temperature and extinction QC value. We also provide advice on the use of extinction QC values to filter data, and end with some general caveats regarding the retrieval of extinction from elastic backscatter lidars. The mathematical details of the changes presented in Sect. 2 are described in the Supplement. This Supplement also includes descriptions of new criteria for determining the existence of solutions, an updated uncertainty analysis, a new initialization of the backscatter solution, and the initialization procedures for retrieving lidar ratios in opaque layers.
The algorithms for retrieving profiles of particulate backscatter, extinction, and optical depth are described in YV09 and Young et al. (2013, 2016, henceforth Y1316). Changes and new features in the analysis parameters and in the algorithms used in the V4 retrievals have led to substantial improvements in these products when compared with those reported in V3. These changes are outlined in the following sections, with the mathematical details appearing in the Supplement. The most significant changes in the analysis parameters are to the initial estimates of particulate lidar ratios and their uncertainties for both aerosols and ice clouds (Sect. 2.1). The multiple scattering factors used for retrieving extinction profiles in ice clouds and in opaque water clouds have also been changed.
The extinction algorithm has also been modified to significantly increase the number of constrained retrievals that are attempted, and the extensive changes made in the treatment of opaque layers now produce much improved extinction profiles in these features (Sect. 2.2). The iterative retrieval of the particulate backscatter at each point in the profile is now initialized in a new way that ensures that the initial value is close to the final value with the combined benefits of reliable and rapid convergence of the iteration. Further modifications to the retrieval algorithms include changes to the assessment of what is and what is not a successful retrieval, and these are reflected in revisions to the extinction QC flags (Sect. 2.2.6).
In this section we summarize the changes introduced in V4 to the initial
lidar ratios for aerosols and clouds. These initial values are used by the
extinction retrieval algorithm for the analysis of semi-transparent layers
where the retrievals cannot be constrained by an independent measurement of
the effective two-way transmittance of the layer. The effective, two-way
layer transmittance,
The relative uncertainties in the initial lidar ratios have also been changed in V4. In most cases these are considerably smaller than in V3. As the lidar ratio uncertainty is usually the dominant factor determining the overall uncertainties in the retrieved extinction data products, these latter uncertainties are expected to be smaller than in V3.
In addition to changes in the lidar ratios and their uncertainties, there are also changes to the multiple scattering factors for clouds. Whereas these factors were constants in V3, in V4 they are now functions of cloud 532 nm attenuated backscatter centroid temperature for ice clouds (Garnier et al., 2015) and are obtained from 532 nm depolarization measurements for opaque water clouds (Hu, 2007).
The initial lidar ratios and their uncertainties for several of the aerosol subtypes have been revised for V4 (Kim et al., 2018). As can be seen in Table 1, V4 specifies larger 532 nm lidar ratios for dust (by 10 %), clean marine (by 15 %), and clean continental (by 51 %) aerosols. These higher lidar ratios, which are consistent with improved knowledge gained over the past decade (e.g., Tesche et al., 2009; Nisantzi et al., 2015; Liu et al., 2015; Papagiannopoulos et al., 2016; Haarig et al., 2017), contribute to higher values in the retrieved backscatter, extinction, and optical depths than were reported in V3. Our better understanding of lidar ratio variability led to significant reductions in the uncertainties ascribed to the lidar ratios for marine, desert dust and smoke aerosols, resulting in generally lower uncertainties in the retrieved products for these aerosol types. As explained in Kim et al. (2018) and documented in the V4 CALIPSO Data Products Catalog (Vaughan et al., 2018c), the nomenclatures of the aerosol type 3 “polluted continental” and type 6 “smoke” were changed in version 4.1 to “polluted continental/smoke” and “elevated smoke”, respectively. V4 also adds a new “dusty marine” tropospheric subtype and five new stratospheric subtypes. The lidar ratio for the dusty marine mixture is significantly higher than that for clean marine, but significantly lower than that for polluted dust. Aerosols identified as dusty marine in V4 were generally classified as polluted dust in V3. Some aerosol lidar ratios at 1064 nm have also been changed, as shown in Table 1. Note that there was no cloud–aerosol classification in the stratosphere in V3 and that all stratospheric layers were assigned an initial lidar ratio of 25 sr and a multiple scattering factor of 1.
Combined with changes in the CAD and aerosol subtyping algorithms that have led to changes in the identification of some aerosol types, these new lidar ratios yield extinction profiles and optical depths that can differ noticeably from those reported in V3. For aerosols, changes to optical depths and extinction profiles are largely the result of changes to the lidar ratios for the particular aerosol subtypes, and of changes in the algorithms that determine these subtypes, and are only rarely due to changes to the extinction algorithm. Therefore, changes in aerosol optical depths are discussed in a separate publication (Kim et al., 2018).
Initial lidar ratios and their uncertainties, in units of steradians, that characterize each of the aerosol subtypes reported in V3 and V4. V4 values that have changed with respect to V3 are in bold text.
n/a: not applicable.
In contrast to V3 and earlier data releases, where the multiple scattering
factors for ice clouds were assigned a constant value of
As a consequence of the changes in the
To ensure consistency between unconstrained and constrained retrievals in
semi-transparent ice clouds, the V4 initial lidar ratios have been derived
from a statistical analysis of nighttime constrained retrievals in clouds
identified with high confidence as being composed of ROI. These constrained
retrievals used direct measurements of effective two-way transmittance,
together with multiple scattering factors generated by the aforementioned
sigmoid approximation function, to retrieve optimized estimates of the V4
initial cirrus cloud lidar ratios. As seen in Fig. 1, the resulting initial
cirrus lidar ratios and the corresponding layer-effective lidar ratios (i.e.,
the product of lidar ratio and multiple scattering factor) both vary as
functions of the layer attenuated backscatter centroid temperature. Like the
multiple scattering factors, the initial cirrus lidar ratios (Fig. 1b) are
approximated by a sigmoid function of backscatter centroid temperature, and
decrease from a value of
CALIOP ice cloud scattering parameters for both V3 and V4 plotted
as functions of cloud 532 nm attenuated backscatter centroid temperature,
showing
For semi-transparent water clouds, there have been no changes to either the initial lidar ratio or the multiple scattering factor. In V4 these have values of 19 sr and 0.6, respectively, as they did in V3. However, the relative uncertainty in the lidar ratio has been reduced from 40 % to 15 %, reflecting substantial improvements in the cloud thermodynamic phase discrimination algorithm (Avery et al., 2018) and the implementation of a new algorithm for estimating multiple scattering factors for opaque water clouds (Hu et al., 2007).
In V3, when the cloud phase could not be identified, a constant lidar ratio of 22 sr was used, this being the average of the ice cloud (25 sr) and water cloud (19 sr) values. In V4, the ice and water lidar ratios are again averaged but now the ice–cloud lidar ratio is obtained from the sigmoid approximation function shown in Fig. 1b. Similarly, the multiple scattering factors in V4 are the average of the value for semi-transparent water clouds (0.6) and the sigmoid function values shown in Fig. 1a. In V3, a constant value of 0.6 was used.
Extinction profile retrievals that are constrained by measurements of the effective two-way transmittance of the layer are generally more accurate than those for which no constraint exists. Constrained retrievals generate optimized values of the layer lidar ratio that are precisely matched to the measured two-way transmittance. However, for unconstrained retrievals, optimized adjustments of the lidar ratio are not possible because no constraint is available, and hence the accuracy of the two-way transmittance estimates derived using this method depends critically on the accuracy of the initial lidar ratio. In V3, constrained retrievals were only performed when the error in the retrieved lidar ratio was estimated to be lower than 40 %. However, Garnier et al. (2015) found good agreement between CALIOP constrained optical depths and IIR optical depths at smaller values of optical depths than implied by the 40 % limit on lidar ratio uncertainty. Indeed, low optical depths correspond to high effective two-way particulate transmittances and, consequently, high lidar ratio uncertainties, as seen in Eq. (7.4) of Liu et al. (2005). By removing the lidar ratio uncertainty threshold in V4, a far greater proportion of constrained retrievals is achieved. Note, however, that the optical depth uncertainty estimates for unconstrained retrievals depend directly on the uncertainties assigned to the initial lidar ratios. In contrast, the optical depth uncertainties for constrained retrievals depend only on the uncertainties in the measured data. The uncertainties in the lidar ratios obtained from constrained solutions likewise depend on the uncertainties in the measured optical depths and are entirely independent of the uncertainties associated with the initial lidar ratios. A paradoxical consequence of this dichotomy is that for features with large uncertainties in their measured optical depths, constrained solutions can generate larger uncertainties in the retrieved backscatter and extinction profiles and derived lidar ratios than would have been the case had an unconstrained retrieval been used instead.
Constrained retrievals use measurements of the effective two-way layer transmittance and its uncertainty that are determined by comparing signals from above and below the layer. The determination of the layer boundaries is detailed in Vaughan et al. (2005, 2009). Briefly, the initial estimate of cloud base is determined as that range at which the attenuated scattering ratio (the ratio of the normalized, range-corrected backscatter signal to a molecular backscatter model) drops below a range-dependent threshold that is determined largely by signal-to-noise ratio (SNR) and signal attenuation by the overlying atmosphere. This initial estimate of cloud base is further refined by continuing to search below this altitude for a region where the attenuated scattering ratio ceases to be a decreasing function of range. Depending on the SNR in the region below the cloud, this refinement sets cloud base below any readily detectable “leakage” of the signal into the assumed clear region caused by, for example, the transient recovery time of the detectors or by multiple scattering from small particles in dense clouds. Note that this multiple scattering leakage is not pulse stretching (Miller and Stephens, 1999), but instead occurs, for instance, because the forward scattering angle from the small ice crystals in cold cirrus can be wider than the CALIOP receiver field of view (Reverdy et al., 2015), so that the multiple scattering factor can be slightly larger below the ice clouds than in cloud (Winker, 2003). Once all features have been detected in a column, so called “clear air” regions above and below each feature are identified. To initiate a constrained retrieval, the V4 CALIOP extinction algorithm requires a minimum feature-free vertical extent of 2.48 km both above and below a candidate feature. The required effective layer two-way transmittance is then calculated as the ratio of the mean attenuated scattering ratios computed over these below-cloud and above-cloud clear air regions. The fidelity of these estimates relies on the supposition that the backscatter signals in the clear air regions are due solely to air molecules. If this condition is not met (and this is impossible to confirm with absolute certainty), then unless the mean particulate scattering ratios in the two clear air regions are identical, the transmittance measurements will be in error, no matter how small the reported uncertainty, and the constrained retrieval will also be in error. These are bias errors, not random errors, as discussed in Sect. 3b2 of Young et al. (2013) and by del Guasta (1998). Undetected particulate layers above the layer being analyzed can also affect the calculated lidar ratio. Extreme errors can cause the derived lidar ratios to approach and sometimes even exceed the physically acceptable limits of 0.05 to 250 sr imposed by the V4 retrieval scheme. Any constrained retrieval (bit 0 set to 1 in the extinction QC flag – See Sect. 2.2.6) in which the derived lidar ratio is equal to either of these limits is to be treated as suspect. When serious errors are encountered in the solution of constrained retrievals, bit 8 is also set, giving an extinction QC flag value of 257.
The criteria for determining whether a retrieval is successful or not have been modified in V4. The equation for calculating the particulate backscatter at a given range is transcendental and is solved iteratively using a Newton–Raphson algorithm (YV09), while the equation for the corresponding uncertainty is now a simple algebraic expression. In V3 it was solved iteratively. (An improved initialization of the backscatter solution used in V4 is provided in Sect. S2 in the Supplement) As shown in Sect. S1, the lidar ratio appears in the equations for both the particulate backscatter and its uncertainty at a given range. It is simple to determine if a given lidar ratio is too large for the current input data, because solutions simply do not exist. In V3, retrieved uncertainties in backscatter, extinction, and optical depth that grew too large were assigned a fixed value of 99.99. However, in V4, the non-existence of an uncertainty solution or the retrieval of an uncertainty that is excessively large are both considered, along with the non-existence of a backscatter solution, to be indications that the lidar ratio is too large. These conditions trigger an algorithm that reduces the lidar ratio. Having the number of iterations to achieve convergence of the backscatter solution exceed some prescribed maximum is also an indication that the lidar ratio is too large.
The sensitivity of extinction and optical depth retrievals to lidar ratio
errors increases dramatically as the layer optical depth increases (Y1316).
In layers determined to be opaque, even small lidar ratio errors can cause
large errors in the retrievals. In V3 and previous versions, the first
retrieval attempts were made using the initial lidar ratio prescribed for
the layer type and subtype. As it is obviously impossible to select an
initial lidar ratio that exactly matches the true value in every actual
layer studied; the initial value is virtually certain to be either too large
or too small. The former case is easily identified, as it is not possible to
retrieve a complete profile if the lidar ratio is too large. In these
situations, the earlier algorithms reduced the lidar ratio, initially in
steps of 1 %, and then in steps of 5 %, until a successful retrieval
was obtained (YV09). However, given the extreme sensitivity of the retrieval
to the lidar ratio and the relatively coarse reductions, the final values of
these reduced lidar ratios may well have been too small. The latter case is
more difficult to identify, as successful (although underestimated)
retrievals using the initial lidar ratio (QC
In order to assess the incidence and likely impact of V3 lidar ratios that
were either too high or too low, we examined all opaque ROI clouds detected
in the V3 data set during January 2010, with separate analyses done for
daytime and nighttime measurements. To estimate the lidar ratio that would be
required for a totally attenuating layer, we use a well-validated method
initially developed by Platt (1973). Given values for the integrated
attenuated particulate backscatter profile through the layer,
For those V3 retrievals where the lidar ratio was not reduced (i.e.,
QC
From this study we draw two conclusions. First, the initial lidar ratio used
for opaque ice clouds in the V3 and earlier data products was, in general,
far too low. As a direct consequence, the retrieved extinction coefficients
and the estimate of optical penetration into the clouds (i.e., the effective
optical depth) were also significantly underestimated. Second, the magnitudes
of the lidar ratio reductions were typically too coarse, resulting in an
underestimate of lidar ratio even in those cases where the lidar ratio was
reduced. Both of these deficiencies have been addressed in the V4 algorithm.
The initial value of the lidar ratio for opaque layers is now obtained using
Eq. (1), with the effective two-way transmittance assumed to be zero. The
multiple scattering factor is derived according to the type of layer being
analyzed. For ice clouds, an initial temperature-dependent multiple
scattering factor is computed based on the same attenuated backscatter
centroid temperature as for semi-transparent clouds (Garnier et al., 2015).
For water clouds, the multiple scattering factor is derived from the
layer-integrated volume depolarization ratio using the method of Hu (2007).
For aerosols, multiple scattering is assumed to be negligible, and hence
For ice clouds, determining the lidar ratio is a two-step process. In the first step, the multiple scattering factor and an initial lidar ratio are calculated based on the temperature at the centroid of the attenuated backscatter profile (Sect. 2.1.2 – Ice clouds). This lidar ratio is recorded in the V4 data products as the initial lidar ratio. After a successful retrieval of the particulate backscatter and extinction over the whole detected depth of the layer, the multiple scattering factor is recalculated at the altitude of the retrieved particulate backscatter centroid. This recalculation is not necessary for semi-transparent layers, where the centroid altitudes of the attenuated backscatter profiles provide reliable approximations of the centroid altitudes of the particulate backscatter profiles. However, for opaque layers, the differences between the two can be large (up to a kilometer or more) and hence the multiple scattering factors at the two altitudes and temperatures can be quite different. As the recalculated centroid height will be at a lower altitude (and likely a warmer temperature) than that of the original attenuated backscatter centroid, the updated multiple scattering factor is likely to be lower than the initial estimate (see Fig. 1a). An updated extinction solution is then calculated using the updated multiple scattering factor and the newly derived estimate of lidar ratio. The optimized lidar ratio obtained from this second solution is recorded in the data products as the final lidar ratio. One consequence of the refinements made to the multiple scattering factor in this two-step solution process is that the final lidar ratio reported in the V4 data products for opaque layers can sometimes be larger than the initial lidar ratio.
It should be noted that the V4 surface detection algorithm (Vaughan et al., 2018b) has led to large improvements in the assessment of what is and what is not an opaque layer and, as a consequence, when the opaque-layer algorithm can be used. Nevertheless, the accuracy of the new algorithm can still sometimes be compromised by the low SNR conditions that can occur, for example, during daytime measurements of bright, opaque clouds. Low SNR may lead both to missed detections by the surface detection scheme and, within the layer detection algorithm, to missed detections of weaker backscatter signals near the top and apparent base of the opaque layer. The former problem causes the layer two-way transmittance to be set incorrectly to zero and the latter causes the calculated layer integrated attenuated backscatter to be too small. Both of these errors lead to an overestimate of the initial lidar ratio. In such cases the retrieved extinction profile and optical depth will likewise both be overestimated.
In Fig. 2, we use simulated data to demonstrate the effect of the different V3 and V4 algorithms on the
retrieved extinction profiles. The top of the simulated cloud layer is at
10 km while the base is at 4 km, although it can be seen in Fig. 2a that
the signal is totally attenuated at an altitude of about 7.5 km. The V3
retrieval (Fig. 2b), using a fixed lidar ratio of 25 sr and multiple
scattering factor of 0.6, can be seen to trend quickly to zero, significantly
underestimating the extinction coefficients at all altitudes. In contrast,
the V4 retrieval, which obtains its lidar ratio from the data, stays much
closer to the target, trending slightly too low from cloud top to
A comparison of the extinction profiles retrieved by the V4 and V3
algorithms in a simulated totally attenuating ice cloud. The simulated layer
has its top at 10 km and its base at 4 km, with a constant extinction
coefficient of 2 km
There are four main potential errors in the inputs supplied to the extinction algorithm. Two of these errors cause the retrieved particulate extinction to be too high, or cause there to be no possible solution at all. The other two cause the extinction to be too low, or even negative.
If the initial lidar ratio estimate is too high, or the correction for the
attenuation caused by overlying layers is too large (i.e., when the optical
depths of overlying layers are overestimated), then the extinction retrieved
in the lower layers will be too high, or, in the worst case, not retrievable
at all. While retrieval problems in the topmost layers in the atmosphere can
usually be attributed to errors in the initial lidar ratios for these
layers, as the retrieval process progresses to lower layers that have an
increasing number of overlying layers, the retrieval problems are
increasingly likely to be at least partially the result of attenuation
correction errors. As there is no completely reliable way of determining
which of these error sources causes an unsuccessful retrieval, unsuccessful
retrievals are addressed by reducing the lidar ratio in both cases.
Sometimes the lidar ratio needs to be reduced to values that are far below
what is considered to be a valid range. These cases are considered to be
failed retrievals and are terminated with the QC flag being set to indicate
the problem. Where V4 retrievals are terminated at an altitude above the
detected base of the feature, a fill value of
When a successful retrieval could not be achieved using a given lidar ratio, the V3 extinction algorithm reduced the lidar ratio by repeatedly applying fixed fractional reductions, initially in steps of 1 % then in steps of 5 %, until either a complete solution was derived or some error condition (e.g., too many solution attempts) occurred [YV09]. The V4 algorithm takes a more nuanced approach. V4 adjustments to the lidar ratio in opaque layers are inversely proportional to the average extinction coefficient retrieved over the path from the top of the layer to the point where the retrieval fails with the current lidar ratio, multiplied by both the retrieved two-way particulate transmittance over the same range and an empirically derived scaling constant. This produces smaller adjustments where the extinction is higher, as is required to avoid overcorrections. Lidar ratio reductions in opaque layers are limited to a maximum of 1 % on each adjustment. For semi-transparent layers, the fractional reduction is 10 % of the relative uncertainty in the initial lidar ratio.
The opposite problem of the retrieved extinction being too small (or even negative) can occur if the initial lidar ratio is too small or the estimated attenuation for overlying layers is too low. As shown in Y1316, when the optical depth of the layer being analyzed is high, these underestimates can introduce significant errors in the extinction retrieval (e.g., as shown in Fig. 2b). However, whereas cases for which the lidar ratio or the overhead attenuation correction is too large are readily identified because the extinction solution will diverge before reaching the bottom of a layer, we know of no reliable metric that can unambiguously identify those instances where the lidar ratio or the overhead attenuation correction is too low. Consequently, unlike the V3 algorithm, the V4 retrieval does not increase the lidar ratio when negative particulate backscatter coefficients are repeatedly retrieved from positive attenuated backscatter inputs.
As in V3, retrieval uncertainties and retrieval errors in overlying layers propagate into and are amplified by extinction retrievals in underlying layers. While the uncertainties reported in the V4 data products correctly account for the effects of estimated random errors in overlying layers, no estimates are provided for possible bias errors that might also be incurred. However, the combined random uncertainty in the optical depths of overlying layers can be used to estimate the bias error introduced by the accumulated attenuation corrections. Estimates of the total bias error in the lower layer can then be obtained using the various equations and figures in Y1316.
Lidar ratio uncertainties are major contributors to the extinction and optical depth uncertainties reported in the data products (Y1316). The initial lidar ratio uncertainties are assigned by the various lidar ratio determination schemes (i.e., as in Table 1). However, there are some circumstances where the initial lidar ratio uncertainty is modified, or even recalculated completely, within the V4 extinction retrieval algorithm. This final value, whether modified or not, is reported in the V4 data products as the final lidar ratio uncertainty and is expressed as an absolute uncertainty in steradians.
Lidar ratio uncertainties are calculated in constrained retrievals from the uncertainties in the measured apparent two-way particulate transmittance of the layer under analysis and the integrated particulate attenuated backscatter of the layer, as in Vaughan et al. (2005). In V4, this uncertainty may be reduced if the transmittance plus or minus its uncertainty exceeds the range [0, 1] or if the calculated lidar ratio plus or minus its uncertainty exceeds the range of acceptable values (currently 0.05 to 250 sr). In opaque features, the lidar ratio uncertainty may also be reduced from the default value using the range of lidar ratios calculated from estimates of the minimum and maximum two-way transmittance of the layer, where the minimum is zero (total attenuation) and the maximum is that value that would produce a particulate backscatter at the detected layer base that is greater than zero where the attenuated backscatter is also greater than zero. If these recalculations produce values that are out of the acceptable range, then the relative uncertainty in the default value is retained. When lidar ratios need to be reduced in order to retrieve a complete profile over the whole depth of the layer, the reported final lidar ratio uncertainty is likewise reduced so as to maintain the relative uncertainty that is calculated either from the default values or from the modified values just described.
The rules described above do not apply for lidar ratio uncertainties for
opaque water clouds because of lidar “pulse stretching”. Pulse stretching
is caused by the broad forward peak of the scattering phase function of the
small spherical droplets typically found in water clouds. The broad forward
peak causes an increase in off-axis scattering in water clouds. With the
increased multiple scattering found at high optical depths (e.g., in opaque
water clouds), this off-axis scattering quickly dominates the backscattered
signal with increasing penetration into the layer. As lidar range information
is measured by the time delay of the backscattered signal, off-axis
scattering causes errors in the measured delay, and thereby causes errors in
cloud penetration depth measurements. Owing to this loss of range information, the particulate backscatter
and extinction uncertainties reported for opaque water clouds in the CALIOP
V4 data products are now assigned a fill value of
In order to accommodate the modifications described above, and other changes
to the algorithms, the extinction QC flags have been modified in V4. The new
values and their explanations are shown in Table 2. The relative occurrence
of each of the QC bits for both clouds and aerosols is provided in Table 3.
It can be seen that while nearly 93 % of the aerosol layers have
identical QC flags in V3 and V4, identical QC values are found in only
65 % of cloud layers. This difference results mainly from the much higher
occurrence of QC
Extinction quality control flags used in version 3 and version 4.
A comparison of the V4 and V3 extinction QC flag values for
identical layers detected during the years 2007–2010. All QC flag values not
shown separately are grouped under QC
We now present the results of the changes described in Sect. 2, beginning with a study of extinction retrievals in tropical clouds in Sect. 3.1. In Sect. 3.2 we present an analysis of lidar ratios retrieved in ice and water clouds using the new opaque layer algorithm (discussed in Sect. 2.2.3). Sect. 3.3.1 compares the optical depths, retrieved using these lidar ratios with the V3 values obtained using fixed values of the multiple scattering factors and initial lidar ratios. Optical depths retrieved in semi-transparent ice clouds using the V4 and V3 algorithms are compared in Sect. 3.3.2, where we demonstrate the superiority of the V4 results by comparing with MODIS C6 data. In Sect. 3.4 we compare V4 and V3 profiles of median extinction coefficient as a function of temperature and extinction QC. We end with comments and advice on the use of extinction QC values for filtering of data in Sect. 3.5 and provide some caveats on using V4 data in Sect. 3.6.
In Figs. 3, 4, and 5, we compare the V3 and V4 532 nm analyses of a complex cloud system composed of opaque ice and deep convective clouds observed over Panama and the adjacent seas by CALIOP on 24 April 2010 around 07:18 UTC. The example highlights the impact of the new opaque-layer extinction retrieval algorithm. Figure 3 concentrates on the retrieved backscatter and extinction coefficients, whereas Fig. 4 is mainly concerned with the uncertainties. The level 1 data were extracted from the files CAL_LID_L1-Standard-V4-10.2010-04-24T07-04-53ZN.hdf and CAL_LID_L1-ValStage1-V3-01.2010-04-24T07-04-53ZN.hdf. The level 2 data were extracted from the 5-km cloud profile files CAL_LID_L2_05kmCPro-Standard-V4-10.2010-04-24T07-04-53ZN.hdf and CAL_LID_L2_05kmCPro-Prov-V3-01.2010-04-24T07-04-53ZN.hdf, and the corresponding vertical feature mask files CAL_LID_L2_VFM-Standard-V4-10.2010-04-24T07-04-53ZN.hdf and CAL_LID_L2_VFM-ValStage1-V3-01.2010-04-24T07-04-53ZN.hdf.
A comparison of CALIOP V4 and V3 532 nm analyses of dense clouds observed over Central America and surrounding seas around 07:18 UTC on 24 April 2010. See text for details.
A comparison of CALIOP V4 and V3 532 nm analyses of dense clouds observed over Central America and surrounding seas around 07:18 UTC on 24 April 2010. See text for details.
The V4 total attenuated backscatter coefficients (scientific data set (SDS)
name
The retrieved particulate backscatter (Total_Backscatter_Coefficient_532) and extinction coefficients (Extinction_Coefficient_532) are plotted in Fig. 3c and e, respectively. Note that, in these products, “Total” refers to the sum of both polarization components of the particulate backscatter, not to the sum of the particulate and molecular components. Figure 3d and f show, respectively, the ratios (V4 divided by V3) of particulate backscatter and extinction coefficients. The differences in the retrieved quantities result not only from the different calibration (Fig. 5a and b) and different molecular density data, but also from the different lidar ratios, plotted in Fig. 3g and h, and multiple scattering factors, plotted in Fig. 3i and j. The V4 lidar ratios differ from the V3 values where the new algorithms have determined that the cloud is opaque, as indicated by the grey colors in the cloud type plots in Fig. 4a and b, and extinction QC values (Extinction_QC_Flag_532) of 16 or 18 in the extinction QC plots in Fig. 4c and d. Where the clouds are not opaque, the new V4 lidar ratios and multiple scattering factors were obtained as described in Sect. 2.1.2 while the previous initial values were used for V3. The difference between the multiple scattering factors in Fig. 3i and j is striking, because a constant value of 0.6 is used for all clouds in V3, whereas the V4 values are dependent on the temperature at the cloud attenuated backscatter centroid. In general, lower and deeper clouds have higher backscatter centroid temperatures and these are associated with lower multiple scattering factors. Conversely, the default V4 lidar ratios are generally lower in the higher and thinner clouds.
Between the latitudes of
The implementation of the opaque layer algorithm in V4 results in generally
higher effective lidar ratios between the latitudes 3 and 10
The uncertainties in the retrieved data products result mainly from random
noise in the attenuated backscatter coefficients (estimated using the noise
scale factors reported in the L1 data products; see Hostetler et al., 2006
and Liu et al., 2006), calibration uncertainties, and uncertainties in the
lidar ratios. (There is also a small contribution from the uncertainties in
the molecular atmosphere model.) V4 calibration uncertainties are documented
in Kar et al. (2018); Getzewich et al. (2018); and Vaughan et al. (2018a).
Lidar ratio uncertainties are discussed earlier in Sect. 2.1.2 and 2.2.5, and
in Kim et al. (2018). As a consequence of these various increases and
decreases in relative uncertainties in V4, and the non-linear nature of the
lidar equation at high optical depths, there is no simple relationship
between the retrieved backscatter and extinction uncertainties in the two
data versions. For example, even though the lidar ratio uncertainties have
been reduced, the generally higher V4 lidar ratios lead to higher retrieved
backscatter and extinction, particularly with increased penetration depth,
and these result in higher relative uncertainties. Focusing on the main ice
cloud layer in Fig. 4e, f, g, and h, in regions where the cloud is not
opaque, the V4 uncertainties are often lower than the V3 values at the top of
the cloud but commonly reach or exceed the V3 values at the base of the
cloud. In the region where the cloud is opaque (latitudes
A comparison of CALIOP V4 and V3 532 nm calibration coefficients
and lidar ratios around 07:18 UTC on 24 April 2010.
The uncertainties in Fig. 4f and h contribute to the uncertainties in the
retrieved cloud optical depths shown in Fig. 4j. These cause higher V4 uncertainties between the latitudes
The new opaque layer analysis permits the retrieval of the particulate lidar
ratio, a parameter of considerable interest. The accuracy of the lidar ratio
depends on the accuracy of the integrated attenuated particulate backscatter
of the layer and of the determination of its opacity. The accuracy of the
first depends on accurate calibration, accurate determination of the
existence of overlying layers and correction for their attenuation, and
accurate determination of the full vertical extent of the layer down to the
altitude at which the backscatter signal is extinguished (i.e., where the
particulate transmittance is truly zero). The accuracy of the second depends
on the reliable determination of complete opacity (particulate transmittance
is zero) by reliably determining the absence of any surface backscatter
signals below the layer. All of these factors are affected by the
signal-to-noise ratio, which is lower during the day, so nighttime
measurements are expected to be to be more reliable. As an example, we
present an analysis of the topmost cloud layers measured during the period
2012–2015 and composed of ROI particles. As seen in Fig. 6a, the
geometric vertical extent is clearly smaller for daytime data (median
Histograms of
To demonstrate that the new algorithm is indeed producing lidar ratios that
are closer to what is expected in opaque clouds than found for V3 retrievals
in Sect. 2.2.3, we repeated our analysis of opaque ROI clouds detected in
January 2010 using the V4 algorithm. For retrievals in which the lidar ratio
was not reduced (QC
In Fig. 7 we present maps of the global variations in 532 nm lidar ratios for opaque ROI clouds measured during daytime (Fig. 7a) and nighttime (Fig. 7b) during the years 2012 to 2015. The data are for the same clouds as used to create the histograms in Fig. 6. Given the large differences in daytime and nighttime calibrations, the excellent agreement shown between the distributions of daytime and nighttime lidar ratios is remarkable and gives confidence both in the calibrations and in the new opaque-layer algorithm. The agreement also suggests that the lidar ratio and, by implication, the microphysics in the uppermost regions of opaque ice clouds, do not change significantly between day and night.
A considerable latitudinal variation can be seen in the lidar ratios with the lowest values found over the tropical region and over Antarctica. If the temperature dependence of the lidar ratio for opaque ROI clouds is the same as that for semi-transparent clouds, this behavior could, at least in part, be explained by the variation in lidar ratio with temperature shown in Fig. 1b. Indeed, in the tropics and over Antarctica, the ice clouds occur at altitudes where the temperature is the coolest, as shown in Fig. 7c and d. There are also some land–ocean differences, which suggest that there could be geographic influences on the cloud microphysics. A more detailed analysis, which is beyond the scope of this present work, would be required to determine the causes of the observed distribution.
Global distribution of lidar ratios measured in opaque ROI clouds
The new opaque layer algorithms described in Sect. 2.2.3 have also been used
to determine the lidar ratio for water clouds. Although retrieval of
extinction profiles in opaque water clouds is of low-to-no confidence (as
indicated by an uncertainty fill value of
Statistical characterization of global measurements of opaque water
clouds during 2008–2010, where the clouds were the only layers detected
in a 5 km averaged column. The integrated attenuated backscatter,
layer-integrated volume depolarization ratio, and layer-integrated
attenuated backscatter color ratio are measured over the detected cloud
geometrical thicknesses. The multiple scattering factors are derived using
the method of Hu et al. (2007), and the lidar ratios are retrieved using Eq. (1). MAD is the median absolute deviation and
Mean global distribution of daytime opaque water clouds during
2008–2010;
We note that the mean daytime lidar ratio of
The mean global distribution of the lidar ratios measured in daytime opaque water clouds during 2008–2010 is shown in Fig. 8. It is immediately apparent that this distribution is quite different from that of the opaque ice clouds shown in Fig. 7. Generally, higher lidar ratios are seen in the Northern Hemisphere compared with these in the Southern Hemisphere. Higher lidar ratios are seen over the land, and adjacent oceanic regions, particularly in the regions influenced by dust in North Africa and the Middle East, Central Asia and China. Regions influenced by smoke in South America and southern Africa also have higher lidar ratios. While it is tempting to explain the higher lidar ratios in these regions in terms of the Twomey effect (Twomey, 1977), where greater numbers of condensation nuclei produce greater numbers of smaller cloud droplets, or by the increase in extinction through the inclusion of absorbing particles within the cloud water droplets (Mishchenko et al., 2014; Chylek and Hallett, 1992; Miles et al., 2000; Wittbom et al., 2014), a more thorough analysis is required before we can reach these conclusions. Apparent increases in lidar ratio can also be caused by SNR limitations, (particularly in the daytime), which appear as undetected overlying aerosol layers, incomplete determination of the full extent of the cloud layer, or incorrect assignment of opacity because of the missed detection of underlying layers or the surface. While a more thorough analysis is beyond the scope of this current paper, the results presented here suggest that further research could be rewarding.
We turn now to the fractional optical
depths retrieved in opaque
clouds using the new methods.
Readers are reminded that the CALIOP optical depths reported for opaque
clouds are retrieved only over the range from the top of the cloud down to
the altitude at which the signal is totally attenuated (i.e., when the signal
is indistinguishable from the ambient noise). Consequently, CALIOP opaque
cloud optical depth estimates cannot be compared with cloud optical depths
measured by passive instruments. In Fig. 9, we present a comparison of the V3
and V4 distributions of optical depths for the topmost opaque (QC
Distributions of the apparent optical depths measured in opaque ROI clouds during 2012–2015 and processed using the V3 and V4 algorithms.
The second notable feature is the large difference in the daytime and
nighttime V4 distributions. This difference is explained by the dramatic
shift from QC
The final feature of note relates to the peaks in the frequency of occurrence of V3 optical depths at around 2.5 and 3.85. When multiplied by a multiple scattering factor of 0.6 that was used for clouds in V3, these values correspond to effective optical depths of 1.5 and 2.5, respectively. The peaks occur because of the fixed reduction steps applied in the V3 algorithms to the lidar ratio, in the case of a solution that would not converge. The peak at an optical depth of 2.5 corresponds to a lidar ratio reduction of 5 %, and that at 3.85 to a reduction of 1 %. These reductions are too coarse in optically dense layers, where the transmittance is close to zero, and cause the artifacts seen in the figure. (See Omar et al., 2009 for a detailed explanation.) These artifacts are not seen in V4 thanks to the refined reduction of the lidar ratios described in Sect. 2.2.4.
Median cloud optical depths (CODs) for semi-transparent, high-confidence nighttime ROI clouds for the period March to May 2010 inclusive, nighttime only. The area-weighted average optical depths from the maps are 0.137 for V3 (77 230 457 samples) and 0.266 for V4 (78 808 302 samples).
Global maps of the nighttime optical depths for semi-transparent high
confidence ROI clouds (QC
As mentioned previously, Holz et al. (2016) showed that while CALIOP V3
constrained (QC
With the changes to the lidar ratios and multiple scattering factors, which, as described in Sect. 2.1, are now functions of temperature, V4 extinction coefficients would be expected to show a rather different height and temperature dependence from that shown by the V3 retrievals. We examine these differences in this section.
A comparison of collocated MODIS C6 optical depths with CALIOP
optical depths retrieved using the V3
In Fig. 12, we compare V3 and V4 cloud extinction coefficients plotted as a
function of temperature. The profiles are the medians of the topmost layers,
identified as clouds with a CAD score of 50 or greater (i.e., classified as
clouds with medium confidence or better), composed of ROI crystals and
detected between 60
For unconstrained retrievals, in which the lidar ratio is unchanged from the
initial value (QC
The large increase in the number of constrained retrievals (QC
The use of higher lidar ratios in V4 would be expected to lead to a greater
number of retrievals in which the lidar ratio needs to be reduced in
semi-transparent clouds (QC
As discussed in earlier sections, the effective lidar ratios used in V3
retrievals in opaque layers were almost always too low. This produced an
underestimate of the retrieved extinction, which increases with penetration
depth, as can be seen the comparison with the V4 profiles shown in Fig. 12h.
Because of the smaller reductions in lidar ratio in response to unsuccessful
retrievals in V4, described in Sect. 2.2.4, the final V4 lidar ratios will
also be larger in QC
Finally, a comment must be made about the kink, or increase in slope, in
almost all of the V3 and V4 extinction profiles at a temperature of
Extinction QC flags are often used to filter data for quality during analyses, and the question is sometimes asked as to which values indicate retrievals that are “trustworthy” or “reliable”. Unfortunately, there is no absolute or straightforward answer to this question and the QC flags should be used while bearing in mind various caveats. That said, only those data with QC flag values of 0, 1, 2, 16, and 18 should be used for reliable scientific analyses.
A comparison of version 4.1 and version 3.3 median ROI cloud extinction profiles measured during September 2014. All clouds were the topmost detected features in the atmospheric column. Daytime and nighttime median extinctions are plotted as functions of temperature and QC value in the column on the right while the corresponding numbers of contributing samples are plotted in the column on the left.
Constrained retrievals (QC
Unconstrained retrievals in semi-transparent layers in which the lidar ratio
has not been reduced (QC
As explained in Sect. 2.2.2, unsuccessful retrievals can occur if either the
input signal or the lidar ratio is too large, or some combination of both.
When this occurs, the lidar ratio is reduced in an attempt to generate a
successful retrieval. Successful retrievals of transparent layers for which
the lidar ratio has been reduced have QC
The initialization of retrievals in opaque layers in V4 is quite different
from that in V3, as the initial lidar ratio is no longer a fixed (albeit
layer-type dependent) value but is instead calculated from the attenuated
backscatter profile. In general, this results in higher initial lidar ratios
in V4, which in turn lead to a vastly increased fraction of opaque-layer
retrievals in which the initial lidar ratio requires reduction (QC
The accuracy of the CALIOP V4 extinction retrievals has been substantially improved relative to previous versions, primarily through the increased use of constrained retrievals and, for unconstrained retrievals, the use of initial lidar ratios that more accurately characterize the identified cloud and aerosol types. Nevertheless, there remain some potential pitfalls in using the CALIOP extinction data products and some areas in which improvements may be possible. In general, these pitfalls are not specific to the CALIPSO lidar or the V4 retrieval scheme, but instead are applicable to the retrieved data products produced by any elastic backscatter lidar.
The fundamental caveat relates to the lidar ratios used in unconstrained
retrievals. Although the V4 initial values are more representative than in
previous versions, natural variability still exists in any aerosol or cloud
type. The magnitudes of the uncertainties ascribed to the CALIOP layer types
are thus largely driven by our knowledge of type-specific natural
variability, so that the downstream uncertainties associated with the optical
profiles reported in the CALIOP V4 data products will generally account for
both noise in the measurements and natural variations in layer properties.
However, our ability to tightly constrain the bounds of this natural
variability in, for example, static regional models is confounded by long
range transport. This is especially so for aerosols such as dusts and smokes,
which show wide variations in lidar ratio (Schuster et al., 2012; Burton et
al., 2012) and are frequently measured in large concentrations far from their
source regions (Uno et al., 2009; Yu et al, 2015; Di Biagio et al., 2018). As
a result, within any single layer, the degree to which the lidar ratio used
in a retrieval differs from the actual value has a direct impact on the
accuracy of the retrieved extinction coefficients and optical depths and this
sensitivity increases with the optical depth of the feature (Y1316). For low
optical depths, the relative error in the retrieved optical depth is closely
approximated by the relative error in the lidar ratio, but for higher optical
depths, the optical depth error increases by a factor of approximately
exp(
There are some circumstances in which constrained retrievals may also be in
error. As explained in Sect. 2.2.1, particulate scattering in regions used
for normalization can lead to biased results. Also, as discussed by Reverdy
et al. (2015), forward scattering from small ice crystals within a cloud can
cause an enhancement in the backscatter signal measured below the cloud that
decreases with range below cloud base. They further suggest that, for CALIOP
signals, the rate of decay is so long that it is only really notable for
cirrus clouds that are composed of small
particles (e.g., 10–20
If multiple scattering induced biases are suspected, the lidar ratios and optical depths where constrained retrievals are employed should be compared with the same parameters in adjacent columns that use unconstrained retrievals. In any case, constrained retrievals in which lidar ratios and optical depths have high relative uncertainties should also be regarded with caution. Finally, in order to assess the likely impact of these potential errors, we refer the reader to the comparison of CALIOP V4 and MODIS C6 optical depths presented in Sect. 3.3.2 and in Fig. 11. The generally very good agreement between the data sets gives a high degree of confidence that the approximations made in the CALIOP analyses have a relatively small impact on the quality of the CALIOP retrievals.
When there are several different layers within a column, the use of an incorrect lidar ratio in layers located higher in the atmosphere results in an error in the retrieved two-way transmittance of that layer. As that transmittance is used to rescale (renormalize) the input profile data used in the retrievals in lower layers, the renormalization error results in retrieval errors in these lower layers (Y1316). For example, if the lidar ratio used for analyzing an upper layer is too low, the retrieved two-way transmittance will be too high. As the attenuated backscatter profiles in underlying regions are divided by this two-way transmittance, the resulting renormalized lower layers will be too weak and the retrievals in these layers too low. In contrast, the use of too high a lidar ratio in upper layers leads to retrievals that are too high in lower layers. These types of errors are cumulative and additional scrutiny should be applied to the optical properties retrieved from layers that underlie several other layers, or beneath appreciable optical depths.
The problem of selecting appropriate lidar ratios can be reduced by increasing the fraction of retrievals that are constrained by two-way transmittance measurements. The use of opaque water clouds (Hu et al., 2007) and/or the Earth's surface (Venkata and Reagan, 2016) as background targets against which the effective two-way transmittance of overlying layers can be estimated is expected to lead to further improvements in extinction retrievals in future versions of the CALIOP data products. But while useful and highly desirable, these full column constraints will not be a panacea, as the information contained in a single constraint must be carefully and correctly parsed out among all the layers detected within a single column.
Finally, we note that the V4 analysis algorithm architecture assumes that a single atmospheric feature detected at a single horizontal resolution has uniform optical properties throughout its vertical and horizontal extent. Any change in particulate type (e.g., from smoke to ice in pyrocumulonimbus) or in the phase of cloud particles (e.g., from randomly oriented to horizontally oriented ice, or from ice to water) will lead to errors in the retrieved profiles and optical depths. For example, the lidar ratio for randomly oriented ice crystals is much higher than that for water droplets and so, in a mixed phase cloud with ice overlying water, the retrieval will most likely fail once the level at which the phase change is reached. The lidar ratio will then be reduced in steps until a successful retrieval is achieved, but the resulting solution will obviously be an artificially homogenized compromise between the correct values.
This paper describes the changes made to the extinction and optical depth retrieval algorithms that are used to create the version 4.10 (V4) CALIOP level 2 data products. The extinction retrievals rely on inputs generated by several algorithms that execute earlier in the analysis process, and thus any V4 changes to these algorithms will also affect the retrieved extinction products to various degrees. Among the most significant of these are changes to the calibration procedures, improvements to the algorithms used for feature finding, cloud–aerosol discrimination, assignment of feature subtype, and extensive updates to the initial lidar ratios and multiple scattering factors. In this paper we concentrate on a description of changes to the extinction retrieval algorithm and the associated input parameters, and only discuss the changes to the other algorithms to the extent that they impact the extinction retrievals. Also, because the changes made to the V4 extinction algorithms mostly affect retrievals in clouds, where the extinction tends to be higher, while aerosol retrievals are more affected by changes to the preceding algorithms and input parameters, we have concentrated on reporting the former and only briefly summarize the changes expected in aerosol layers.
The main input parameters to the extinction retrieval process are the lidar ratios and multiple scattering factors for the particles in the features being analyzed. As discussed in Sect. 2.1.1 and presented in Table 1, there have been changes to the initial lidar ratios and their uncertainties for several of the aerosol types. In particular, there have been increases to the 532 nm lidar ratios for the clean marine (15 %), desert dust (10 %), and clean continental (51 %) aerosol types. In addition, a new dusty marine aerosol type has been added with a lidar ratio that is significantly higher than the clean marine type, but significantly lower than the polluted dust type. Absent any other changes, these higher lidar ratios will lead to higher values of retrieved particulate extinction. Similarly, reductions in the relative uncertainties ascribed to these lidar ratios will reduce the relative uncertainties in the retrieved extinction coefficients and optical depths. The new values of the aerosol lidar ratios in V4 have most often been derived from careful comparisons of CALIOP retrievals to measurements made by airborne HSRL along the same ground track and at around the time of the CALIOP overpass (Rogers et al., 2014; Burton et al., 2013). Use of these improved lidar ratios in retrieving the V4 CALIOP retrievals will produce results that are more representative of actual conditions than in previous data releases.
Among the changes to the extinction retrieval algorithms are revised criteria for determining the acceptability of a retrieval at a particular range step (height), discussed in Sect. 2.2.2. The algebraic solution for backscatter uncertainty (Sect. S1.2) used in V4 now indicates when the current lidar ratio is too large for a solution to exist when used with the current input signal. An excessive number of attempts at a retrieval at a given range is now also interpreted as the use of a lidar ratio that is too large. In both cases, the lidar ratio is reduced and the retrieval restarted from the top of the feature. These changes are reflected in changes to the extinction QC flag values, as discussed in Sect. 2.2.6.
In the previous versions of the extinction retrieval algorithms, when the lidar ratio was determined to be too high to permit a solution and required reduction, it was reduced in steps, initially of 1 %, then of 5 %. As the sensitivity of retrieved extinction profiles and optical depths increases rapidly as the optical depth increases (e.g., Y1316), a reduction of this magnitude may well be too large for features with high optical depths, especially those that are determined to be opaque. This concern was supported by a study of opaque ROI clouds detected in January 2010. In V4, the algorithm for reducing the lidar ratio to achieve a solution has been changed so that it is smaller in optically thick features and larger in more tenuous features. (See Sect. 2.2.4.)
The most significant change to the extinction retrieval algorithms has been the introduction of a new algorithm for retrievals in opaque features discussed in Sect. 2.2.3. Whereas previous versions used a fixed value of lidar ratio for each feature type, the initial lidar ratio in V4 is now derived from CALIOP measurements, namely the integral of the layer attenuated backscatter signal. Should this value be too large to permit a retrieval, V4 implements a very conservative lidar ratio reduction scheme that produces more accurate extinction profiles. In particular, the large underestimates that frequently occurred in V3 have been eliminated (e.g., see Fig. 9).
It is quite difficult to verify these improvements by comparing with observations from other instruments. The lidar signal does not sample the whole depth of opaque clouds so cannot produce a full layer optical depth that could be compared (after scaling for wavelength differences) with passive instruments, which only produce an optical depth and not an extinction profile that could be compared with CALIOP's. Also, because the range of particle sizes in clouds, particularly near cloud tops, can span both the Rayleigh and Mie scattering ranges for radars, whereas the particles are very large compared with CALIOP's wavelength and are totally within the Mie scattering range, the shapes of CALIOP's extinction profiles are likely to be different from those retrieved by the Cloud Profiling Radar onboard the CloudSat satellite. Despite this lack of independent verification, evidence that the new opaque layer algorithm, with its new initialization of the lidar ratio and refined lidar ratio reduction scheme, has indeed led to improved retrievals was provided in Sect. 3.2.1 by a comparison of V4 and V3 retrievals of lidar ratios in the topmost ROI cloud layers detected at night during January 2010.
The newly developed surface detection scheme implemented in V4 markedly improves CALIOP's surface detection performance, and thus permits the use of the opaque layer algorithm with considerable confidence. Using layer integrated attenuated backscatter to calculate the initial lidar ratio in a feature that is wrongly identified as being opaque will result in lidar ratios that are too high with the result that an inaccurate extinction profile could be retrieved, depending on whether or not a lidar ratio reduction was triggered. The new surface detection algorithm ensures that such errors occur much less frequently than in the V3 data set, especially at nighttime when the signal-to-noise ratio is high and both the surface and the full vertical extent of the feature can be determined with high accuracy.
A second major improvement in the retrieval of extinction profiles and optical depths in semi-transparent ice clouds was achieved by incorporating a significant advance in the parameterization of ice cloud multiple scattering obtained from in-depth statistical analyses of several years of CALIPSO measurements. Using an extensive data set of CALIOP measurements of semi-transparent clouds collocated with CALIPSO imaging infrared radiometer optical depth retrievals, Garnier et al. (2015) derived temperature-dependent multiple scattering factors and a corresponding temperature-dependent parameterization of initial cirrus lidar ratios. These new characterizations of light scattering in cirrus clouds have led to significant changes to the magnitude and height dependence of the extinction profiles in these clouds, as shown in Sect. 3.4. Section 3.3.2 presents a comparison of the optical depths reported in the MODIS collection 6 and CALIOP V4 data products for collocated semi-transparent ice clouds measured over the ocean during the day. The excellent agreement shown in Fig. 11 demonstrates the substantial improvements made by adopting the new lidar ratios and multiple scattering factors employed in the V4 retrieval algorithms and gives confidence in the new analysis scheme.
The following CALIPSO standard data products were used in
this study: the CALIPSO level 1 profile product (Vaughan et al., 2018c; NASA
Langley Research Center Atmospheric Science Data Center;
The supplement related to this article is available online at:
SY and MV developed the extinction retrieval algorithms and led the program of test runs of the various development versions with considerable assistance from JL.
The extensive suite of test data, which was designed to simulate actual CALIOP data expected from various types of atmospheric scenes and used for developing and testing the extinction algorithms, was prepared by KP.
The new temperature-dependent multiple scattering factor and lidar ratios for cirrus clouds were developed by AG and MV.
Data analysis for the paper was performed by MV, JT and SY, who also prepared the figures with contributions from AG.
SY led the writing of the manuscript with considerable contributions from MV and AG.
The authors declare that they have no conflict of interest.
This article is part of the special issue “CALIPSO version 4 algorithms and data products”. It is not associated with a conference.
The work described in this paper was supported by NASA Langley Research Center Contract NNL16AA05C (STARSS-III). The authors thank Jay Kar, whose close examination of our manuscript helped us eliminate some subtle, but important, misstatements. Edited by: Vassilis Amiridis Reviewed by: three anonymous referees