2.1 OCO-2 radiative transfer calculations

We use the OCO-2 Level 2 Full Physics Radiative Transfer Model (L2RTM) that was developed for the OCO-2 XCO₂ retrieval. Associated wrapper code handles inputs such as interpolated ECMWF meteorological fields and accounts for the OCO-2 satellite orbit, viewing geometry and instrumental response as described in the OCO-2 data version 6 documentation (Boesch et al., 2015). The radiative transfer is based on the LIDORT radiative transfer model with a correction for the first two orders of scattering (Natraj and Spurr, 2007; Spurr, 2006; Spurr et al., 2001) that fundamentally follows the eigenvector approach to solving the radiative transfer equation (Flatau and Stephens, 1988). This model accounts for Earth's curvature for calculating atmospheric path length of the incident and reflected solar beam but is otherwise horizontally homogeneous. More details are provided in Spurr (2006) and O'Dell (2010).

Although the L2RTM was designed for clear-sky XCO₂ retrievals, it has been validated in cloudy atmospheres by comparing OCO-2 observations with L2RTM output assuming collocated MODIS and CALIPSO cloud properties (Richardson et al., 2017). For homogeneous single-layer liquid clouds over ocean, the root mean square error (RMSE) in continuum channels was ±18 %, an overestimate of the model-only error as this includes 3-D cloud effects, collocation error, parallax effects and uncertainty in the MODIS and CALIPSO retrievals.

Clouds are implemented as follows: the atmosphere is defined on 20 levels, of which one is defined as the cloud centre, one as the cloud top and one as the cloud bottom. The cloud top is placed at the cloud-top pressure and the other cloud levels are equidistantly spaced to cover the cloud pressure thickness. An extinction coefficient is assigned to the centre level to result in the desired optical depth. Above the cloud the pressure levels are linearly interpolated from the cloud top to 1 Pa. Below the cloud they are linearly interpolated from the cloud bottom to the surface pressure. The level selected for the cloud centre is that whose pressure is closest to the cloud centre when linearly interpolated across the 20 levels from the surface pressure to 1 Pa. The L2RTM assigns extinction coefficients to layers by interpolating between levels, so a vertically homogeneous cloud layer is assumed.

Mie scattering computations are used within clouds using relevant coefficients that are pre-calculated for gamma distributions of cloud droplets based on a summary of low-cloud studies (Miles et al., 2000). These values have only been pre-computed for integer values of effective droplet size. This should not affect our results greatly since our calculated uncertainties include a term spanning a range of droplet sizes. Water surfaces at nadir are dark, and even in cloud-free cases there is rarely sufficient SNR for the OCO-2 algorithm to attempt an XCO₂ retrieval. We assume a Cox–Munk surface reflectance function with the L2RTM surface reflectance set to 0.10, but as we only use nadir view over ocean there is little sensitivity to surface properties.

2.2 Optimal estimation and information content

We follow the principles of optimal estimation from Rodgers (2000), where a Bayesian retrieval combines an observation vector y with a prior state vector x_a and obtains a posterior state $\widehat{\mathit{x}}$. In our case the state vector consists of cloud-top pressure P_top, cloud pressure thickness ΔP_c and cloud optical depth τ. This assumes that the observation can be related to the state by a linear forward model with some error ϵ:

where we refer to K as the Jacobian matrix as its elements are $K_{i,j}=\partial y_i/\partial x_j$. Assuming Gaussian distributions associated with x_a and y, Rodgers (2000) shows that the best estimate of the posterior state is

and its covariance matrix is

Here S_a is the prior covariance and S_ϵ the observation covariance. From Eq. (2) the posterior state $\widehat{\mathit{x}}$ is the prior x_a plus an iteration that is based on the difference between the observed and expected y with appropriate weighting for uncertainties. Equation (3) shows that the posterior uncertainty $\widehat{\mathbf{S}}$ is reduced by an amount that depends on the size of the Jacobian K weighted by the observation uncertainty S_ϵ. Potential non-linearity in y(x) is addressed by iteration, with the linear expansion being determined about each iteration step.

In our OCO-2 cloud retrieval the state vector contains optical depth, cloud pressure thickness and cloud-top pressure while the observation vector is any subset of the 853 valid OCO-2 A-band channels. Using fewer channels reduces the computational burden, both in terms of the radiative transfer and for iterating the retrieval which would otherwise involve repeated inversion of 853 × 853 matrices.

It is common practice to select channels based on information content and/or degrees of freedom for signal (Chang et al., 2017; Mahfouf et al., 2015; Martinet et al., 2014; Rabier et al., 2002), and this approach has already been used in an oxygen A-band and B-band analysis for aerosol retrievals (Ding et al., 2016).

The information content is based on the concept of Shannon entropy and is related to the volume of state space occupied by the probability distribution P that represents our knowledge:

It is expressed in bits, which represents the number of binary digits required to represent the possible outcomes. A retrieval decreases the probability distribution volume, and this change in associated Shannon entropy (Shannon and Weaver, 1949) is the information content, IC, of the measurements:

In this case S(P₀) is the Shannon entropy associated with the original probability distribution and S(P₀) the same value associated with the retrieved probability distribution. For multivariate Gaussian descriptions of the probability distributions, Rodgers (2000) shows that the information content of measurements is

A related property is the degrees of freedom for signal d_s, which represents the number of useful independent quantities in a measurement. It may be thought of as how many different variables can be obtained from a measurement, and with our three-component state vector we require a value approaching three. It may be calculated from the prior and posterior state covariances as follows:

Note the different order and inversion state of the covariance matrices relative to Eq. (6). In our analysis we calculate IC, d_s and posterior errors for continuous micro-windows of varying size and these calculations require S_ϵ and S_a. We assume prior covariances based partially on a MODIS and CALIPSO cross-validation (Richardson et al., 2017) and calculate the observation covariance S_ϵ by perturbing atmospheric profiles. The calculation of the covariances is described in Sect. 3.1 and the channel selection approach in Sect. 3.2.

While theoretically three channels is sufficient to retrieve three state vector elements, it is not clear that the same three channels will apply in all cases. For example, while changes in cloud-top pressure of higher clouds may lead to strong responses in channels near line cores, light in these channels may be mostly absorbed by the time it reaches lower clouds, so less strongly absorbing channels will be preferred for lower clouds. Changes in absorption due to temperature or water vapour may also affect the relative response of radiances to cloud properties. For this purpose, we consider a variety of atmospheric and cloud properties.

Necessary observation covariances are derived by perturbing atmospheric profiles and the IC, and d_s and posterior covariance are used to select an optimal micro-window. Finally a retrieval is developed and tested on cloudy atmospheres where the “truth” is assigned and pseudo-observations and prior values are provided by sampling from the previously defined covariance matrices.

3.1 Calculation of observation covariances

For simplicity we assume that the components of S_ϵ are independent and consider error contributions from instrumental uncertainty S_I and that introduced by uncertainty in the temperature profile S_T, humidity profile S_q and effective droplet radius S_reff such that

In reality, the temperature and humidity uncertainties are likely to be correlated, but this simplifies the calculation and allows unique attribution of covariance sources. The matrix S_I is a diagonal matrix, so averaging over more channels reduces the total posterior uncertainty even if the Jacobians are not independent. Its elements are equal to the square of the instrumental uncertainty, which depends on the radiance.

For S_T and S_q we follow the approach of Chang et al. (2017) and perturb the tropical, mid-latitude and high-latitude atmospheric profiles 2000 times for temperature and humidity separately with uncertainties based on 1 km resolution AIRS validation results (Divakarla et al., 2006). For temperature we add a uniform perturbation to each level with a value sampled from a zero mean (μ) Gaussian distribution with standard deviation (σ) of ±1.5 K. For specific humidity we sample from a zero mean Gaussian distribution with a standard deviation of unity, then scale this value based on pressure level. The scaling is equivalent to ±20 % of the initial specific humidity at the surface, increasing linearly to ±50 % of the layer values at 250 hPa and remaining at ±50 % for levels with lower pressure. The calculation was also performed with 2000 perturbations applied to r_eff by sampling from a lognormal distribution that approximates the effective radius distribution reported by MODIS for our November 2015 low cloud cases. This lognormal fit has an arithmetic mean of 12.0 µm, but after excluding values outside the 4–30 µm retrieved by MODIS, the arithmetic mean is 12.6 µm and 5–95 % of the values fall within 7.5–19.4 µm. We choose r_eff = 12 µm in our default retrieval as we are restricted to integer values by the available L2RTM Mie scattering tables, and based on its similarity to the full distribution mean.

For each set of perturbations we simulated the A-band spectra for cloud optical depths of 5, 10 and 25 and solar zenith angles (SZAs) of approximately 30, 45 and 60^∘ with a cloud-top pressure of 850 hPa. We calculate covariances at a single value of P_top, but the convergence of our synthetic retrieval tests across a range of true P_top values shows that we obtain reliable results regardless.

The output spectra are provided for each of the eight different instrument line shapes associated with the eight different OCO-2 across-track sounding positions.

For each set of 2000 perturbed outputs, we estimated the covariance matrix elements, S_i,j where i, j refer to channel indices, as follows:

where the sum is over the N = 2000 spectra of radiance I, which are individually referred to using the index k. In this case $<I_i>$ and $<I_j>$ are the sample mean radiances in the relevant channels i and j.

3.2 Channel selection

Equations (3) and (6) state that we can determine the information content and posterior error covariance from the prior covariance, observation covariance and Jacobians. Our aim is to select the optimal micro-window of consecutive OCO-2 channels to provide a retrieval that efficiently reduces the posterior state error.

We use the L2FP radiative transfer model to simulate OCO-2 spectra for marine liquid clouds of τ at 5, 10, 25 and P_top at 680, 750, 850 hPa, for each of the three meteorological cases described in Sect. 3.1 and for each of the eight across-track sounding positions. In each case, the solar zenith angle is 45^∘ and the Jacobians for τ, P_top and ΔP are determined by finite differencing. The relevant observation covariance is that determined for the same sounding position, region and optical depth in Sect. 2.2 at SZA = 45^∘. Prior covariance is assumed to be diagonal, equivalent to an error of 1.5 in τ, of 60 hPa in P_top and of 7.5 hPa in ΔP. Our τ prior error comes from applying the ±18 % error in simulated radiance for homogeneous clouds when provided with MODIS optical depth (Richardson et al., 2017). Our P_top uncertainty is from the standard deviation of the differences between OCO-2 and CALIPSO P_top when using a simple lookup table for OCO-2, which we intend to use for the OCO-2 prior. The ΔP uncertainty is similar to the ±20 % error associated with Eq. (8) for clouds of cloud fraction > 0.8 reported in Bennartz (2007).

We consider the information content IC, and the three diagonal elements of the posterior covariance matrix S_x. The information content accounts for non-diagonal terms in the posterior covariance, allowing an objective best selection, while the diagonal elements allow more intuitive interpretation of the magnitude of the posterior uncertainty. We refer to these using the symbol σ with a relevant subscript, such that ${\mathit{\sigma }}_{\mathit{\tau }}^{\mathrm{2}}=S_{\mathit{\tau },\mathit{\tau }}$, where S_τ,τ is the element of the covariance matrix corresponding to the τ−τ covariance. Note that we present the square root of this value, i.e. σ.

This approach represents a sample of 27 unique cloud–meteorology cases across the eight different sets of OCO-2 instrument line shapes, resulting in 216 total cases. When selecting the optimal micro-window for retrievals, it is necessary to select not just its location but also its size (i.e. number of neighbouring channels within the micro-window).

To make this problem tractable, we select micro-windows of the following size: 5, 10, 25, 50, 75, 100, 150, 200 and 500 neighbouring channels. For each of these possible sizes we calculate IC, d_s and the diagonal posterior error terms for every overlapping micro-window of that size. For example, the 853 individual OCO-2 channels allow 849 overlapping five-channel micro-windows, for which we determine the information content values for each of the 216 cases.

For each size of micro-window we choose the one with the highest mean information content across the 216 cases. While this may result in a different location for each size of micro-window, the location is fixed for an individual case; i.e. the five-channel micro-window consists of the same five channels in all 216 cases. We select the optimal micro-window size as that with > 80 % of the 500-channel IC, optical depth posterior σ_τ,τ better than ±0.05 and a posterior of better than ±1 hPa in the pressure terms ${\mathit{\sigma }}_{P_{\mathrm{top}},P_{\mathrm{top}}}$ and σ_ΔP,ΔP for all 216 cases. These thresholds are by nature subjective and arbitrary.

3.3 Theoretical retrieval test case

We perform synthetic retrievals with known true cloud cases in mid-latitude meteorology and a 45^∘ solar zenith angle. For each cloud case we perform 50 retrievals using a 12 µm droplet size and the prior cloud state is sampled from Gaussian distributions with σ_τ of ±30 %, ${\mathit{\sigma }}_{P_{\mathrm{top}}}$ of ±60 hPa. Cloud pressure thickness is calculated from Eq. (8) with LWP from Eq. (9), and in the optimal estimation a prior σ_ΔP of ±25 % is assumed. The atmospheric humidity and temperature profiles are perturbed by sampling from the same distributions used to derive the covariance matrices in Sect. 3.2 and the observed spectrum in each case is generated by taking the simulated spectrum from the “truth” case and perturbing it by sampling from the relevant covariance matrix that has been scaled for the cloud properties according to Sect. 3.2. The squared OCO-2 radiance uncertainties are added to the diagonal elements of the observation error covariance matrix with no cross correlation. We use the standard OCO-2 version 7 uncertainties, and SNR increases as the radiance in a given channel increases. The median SNR for an individual spectrum ranges from just over 400 for the τ = 5 cases to around 700 for the τ = 25 cases. The single-channel SNR reaches a minimum of 72 in an absorption band channel in a τ = 5 case, and a maximum of 763 in a weakly absorbing channel in a τ = 25 case.

Forty true cloud cases are used with five of each case where optical depth ranges from 5 to 40 in increments of 5 and cloud-top pressure is randomly selected to be between 680 and 900 hPa and rounded to the nearest 10 hPa. The prior cloud properties are assumed to be unbiased and so are randomly sampled from a Gaussian distribution with a mean equal to the truth and a standard deviation equal to the prior errors above. Each synthetic retrieval begins with a separate prior, and the prior is also used as the first guess. The retrieval attempts assume r_eff = 12 µm but the true r_eff is allowed to vary and is randomly sampled from a literature summary of marine stratocumulus results, scaled to ensure a mean value of 12 µm (Miles et al., 2000). The r_eff distribution effective variance is fixed in each case in order to use the pre-calculated scattering properties used with the L2RTM code, but given the wide range of effective mean values considered, it is not expected that allowing the effective variance to change would greatly affect the results.

For each of the 50 perturbed prior states and observation spectra, we perform a standard 10-iteration optimal estimation retrieval (Rodgers, 2000) using the Gauss–Newton solution to optimise each step. These retrievals are done using the 75-channel micro-window selected following Sect. 3.3. The sample means and standard deviations are then compared with the known true state and indicate the theoretical performance of the micro-window retrieval.

4.1 Micro-window selection

Figure 4 shows IC and d_s spectra using micro-windows consisting of 5, 75 or 200 OCO-2 channels. Also shown are the posterior errors in cloud properties taken from the square roots of the diagonal components of S_x.

Figure 3Two-dimensional histograms of covariance matrix elements at 60^∘ solar zenith angle as a function of the same values at 30^∘ solar zenith angle for the mid-latitude meteorological state; each value has been scaled by ${\mathit{\mu }}_{\mathrm{0}}^{-\mathrm{2}}={\cos }^{-\mathrm{2}}\left(\mathrm{SZA}\right)$ to account for differences in illumination geometry.

Download

Figure 4Results of information content analysis for a τ = 10 and P_top = 850 hPa cloud in mean mid-latitude meteorology for OCO-2 sounding position 1. Each line represents the result using a micro-window of difference size, centred on the OCO-2 channel given in the x axis. Values are as follows: (a) information content in bits and (b) degrees of freedom for signal. Panels (c–e) show the square root of diagonal elements of posterior state covariance matrix: (c) cloud optical depth, (d) cloud-top pressure and (e) cloud pressure thickness.

Download

In this cloud case (mid-latitude, τ = 10, P_top = 850 hPa), the greatest information content comes from selecting channels near absorption features and avoiding the far wings of the A-band where only optical depth is reliably retrieved, as these channels have little O₂ absorption and so are uninformative about photon path length. Otherwise, the five-channel micro-window is most sensitive to its placement within the spectrum: information content varies from 4.4 to 9.4 bit depending on the micro-window's location.

Figure 5Range of performance for best-located micro-window of each size. The point represents the central value and the lines the full range of the 216 outputs covering each sounding position, meteorology and cloud case. Note that the x axis is non-linear. (a) Information content in bits and (b) degrees of freedom for signal. Panels (c–e) show square root of diagonal covariance matrix elements: (c) cloud optical depth, (d) cloud-top pressure and (e) cloud pressure thickness. Dashed lines represent selected retrieval requirements.

Download

Micro-windows that contain fewer channels are more sensitive to changes in the instrument line shapes and cloud conditions. For example, for the five-channel micro-window in Fig. 4, the best-performing channel has an information content of 9.4 bit. However, for a different cloudy case, τ = 25, P_top = 680 hPa, and for sounding position 8 instead of 1, the information content is reduced to 6.0 bit. This is a substantial loss relative to the best possible micro-window for that cloud case, which has 8.4 bit of information.

To assess the relative trade-offs between increased speed and decreased performance we take the micro-window with the highest mean information content across all cases. We then plot the central value and full range of the 216 values for each selected micro-window size in Fig. 5, along with our chosen thresholds as dashed lines in each panel. The median case in the 50-channel micro-window passes our IC threshold and in all cases passes the τ uncertainty threshold, but it has multiple cases that fail the P_top and ΔP_c thresholds. By contrast, the 75-channel micro-window containing the OCO-2 channels 353–426 (indices counting from one for the full 1016 OCO-2 L1bSc channels) consistently satisfies our P_top and ΔP_c criteria and reduces the full wavelength range from 759.2–771.8 to 763.5–764.6 nm.

Figure 6 shows an example cloudy scene spectrum simulated for OCO-2 and highlights the chosen 75-channel micro-window in red. Also shown is an approximated GOME-2 spectrum based on the MetOp-B instrument characteristics (Munro et al., 2016). We approximate the ILS using Gaussian instrument line shapes, taking the 0.21 nm spectral sampling from Table 1 and FWHM of 0.50 nm from Table 2 of Munro et al. (2016). While OCO-2 spectra allow three independent pieces of information to be obtained (see the reported d_s in the figure caption) our calculations agree with previous work that the GOME-2 resolution only provides approximately two (Schuessler et al., 2014). Consistent with older theoretical work (Heidinger and Stephens, 2000; O'Brien and Mitchell, 1992) this analysis supports the case the high spectral resolution of OCO-2 leads to additional information about cloud geometric thickness.

4.2 Theoretical retrieval case

Example synthetic retrieval iterations using the 75-channel micro-window are shown in Fig. 7 for τ=10 and τ=25 cases, and convergence typically occurs within a few iterations. Lines are coloured according to their χ² values and it is clear that these are larger for cases where the result settles away from the true state. The posterior sample standard deviations are presented in Table 1 for the full samples and for cases where we filter the results by excluding the 10 % of cases with the highest χ² in each case. The greatest effect of filtering by χ² is to reduce the uncertainty in the cloud-top pressure and cloud pressure thickness from 12.9 to 2.9 and 2.5 hPa respectively. The mean standard deviation in the τ retrieval is ±0.75 across all cases, but this is inflated by a large value in the τ = 35 cases.

Figure 6Example simulated cloudy scene A-band spectrum, for a τ = 10, P_top = 850 hPa cloud in a tropical atmosphere with a solar zenith angle of 45^∘. The black line shows the full OCO-2 simulated spectrum, the blue line is the black line resampled using approximate GOME-2 instrument line shapes and the red line is the selected 75-channel micro-window for OCO-2 cloud retrievals. The legend also reports the d_s for each spectrum with the GOME-2 instrumental uncertainty based on an SNR of 100 as in previous work (Schuessler et al., 2014).

Download

Table 1Posterior errors estimated from the sample standard deviations of the retrieval output. In each case, σ refers to the full sample standard deviation and σ (filtered) refers to the sample standard deviation excluding those with the 10 % highest values of χ². The bottom row shows the mean of the standard deviations entered in each row.

Download Print Version | Download XLSX

Figure 7Example iteration in retrieved cloud properties for synthetic micro-window retrievals for test cases with cloud optical depth near 10 (a) and 25 (b). Each line represents the iterations through 1 of the 50 sample retrievals. The lines are coloured according to their χ² values; note the separate colour bars for the top and bottom rows with larger χ² for the top cases.

Download