Carbon monoxide column retrieval for clear-sky and cloudy atmospheres : a full-mission data set from SCIAMACHY 2 . 3 μ m reflectance measurements

We discuss the retrieval of carbon monoxide (CO) vertical column densities from clear-sky and cloud contaminated 2311–2338 nm reflectance spectra measured by the Scanning Imaging Absorption Spectrometer for Atmospheric Chartography (SCIAMACHY) from January 2003 until the end of the mission in April 2012. These data were processed with the Shortwave Infrared CO Retrieval algorithm (SICOR) that we developed for the operational data processing of the Tropospheric Monitoring Instrument (TROPOMI) that will be launched on ESA’s Sentinel-5 Precursor (S5P) mission. This study complements previous work that was limited to clear-sky observations over land. Over the oceans, CO is estimated from cloudy-sky measurements only, which is an important addition to the SCIAMACHY clear-sky CO data set as shown by NDACC and TCCON measurements at coastal sites. For Ny-Ålesund, Lauder, Mauna Loa and Reunion, a validation of SCIAMACHY clear-sky retrievals is not meaningful because of the high retrieval noise and the few collocations at these sites. The situation improves significantly when considering cloudy-sky observations, where we find a low mean bias b =±6.0 ppb and a strong correlation between the validation and the SCIAMACHY results with a mean Pearson correlation coefficient r = 0.7. Also for land observations, cloudy-sky CO retrievals present an interesting complement to the clear-sky data set. For example, at the cities Tehran and Beijing the agreement of SCIAMACHY clear-sky CO observations with MOZAIC/IAGOS airborne measurements is poor with a mean bias of b = 171.2 ppb and 57.9 ppb because of local CO pollution, which cannot be captured by SCIAMACHY. For cloudy-sky retrievals, the validation improves significantly. Here the retrieved column is mainly sensitive to CO above the cloud and so not affected by the strong local surface emissions. Adjusting the MOZAIC/IAGOS measurements to the vertical sensitivity of the retrieval, the mean bias adds up to b = 52.3 ppb and 5.0 ppb for Tehran and Beijing. At the less urbanised region around the airport Windhoek, local CO pollution is less prominent and so MOZAIC/IAGOS measurements agree well with SCIAMACHY clear-sky retrievals with a mean bias of b = 15.5 ppb, but can be even further improved for cloudy SCIAMACHY observations with a mean bias of b = 0.2 ppb. Overall the cloudy-sky CO retrievals from SCIAMACHY short-wave infrared measurements present a major extension of the clear-sky-only data set, which more than triples the amount of data and adds unique observations over the oceans. Moreover, the study represents the first application of the S5P algorithm for operational CO data processing on cloudy observations prior to the launch of the S5P mission.

is mainly sensitive to CO above the cloud and so not affected by the strong local surface emissions.Adjusting the MOZAIC/IAGOS measurements to the vertical sensitivity of the retrieval, the mean bias adds up to b = 52.3ppb and 5.0 ppb for Tehran and Beijing.At the less urbanised region around the airport Windhoek, local CO pollution is less prominent and so MOZAIC/IAGOS measurements agree well with SCIAMACHY clear-sky retrievals with a mean bias of b = 15.5 ppb, but can be even further improved for cloudy SCIAMACHY observations with a mean bias of b = 0.2 ppb.Overall the cloudy-sky CO retrievals from SCIA-MACHY short-wave infrared measurements present a major extension of the clear-sky-only data set, which more than triples the amount of data and adds unique observations over the oceans.Moreover, the study represents the first application of the S5P algorithm for operational CO data processing on cloudy observations prior to the launch of the S5P mission.

Introduction
The Tropospheric Monitoring Instrument (TROPOMI) will be launched on board the Copernicus Sentinel-5 Precursor (S5P) satellite.Besides the ultraviolet, visible and near infrared spectral range, it will measure the Earth's radiance and solar irradiance in the 2.3 µm short-wave infrared (SWIR) spectral range during its expected lifetime of 7 years (Veefkind et al., 2012).The TROPOMI SWIR spectrometer builds upon the Scanning Imaging Absorption Spectrometer for Atmospheric Chartography (SCIAMACHY) that was T. Borsdorff et al.: SCIAMACHY CO retrievals for clear-sky and cloudy atmospheres operational from January 2003 to April 2012 on ESA's EN-VISAT satellite (Bovensmann et al., 1999).The SWIR measurements of TROPOMI and SCIAMACHY will be similar with respect to spectral coverage and resolution, but TROPOMI will have better spatial resolution (7 × 7 km 2 ), a larger signal-to-noise ratio and daily global coverage.Therefore, TROPOMI will extend the record of SWIR measurements from space, and because of its similarity to SCIA-MACHY, it is possible to use the same CO retrieval algorithm for both instruments and thereby provide a consistent data product for a long-term study of CO (Borsdorff et al., 2016;Landgraf et al., 2016b).
A major limitation of the SCIAMACHY CO data set is its high retrieval noise, which can exceed 100 % of the retrieved value for individual columns.Hence, in practice the data need to be averaged spatially and temporally to reduce the noise contribution (de Laat et al., 2007;Gloudemans et al., 2006).The SCIAMACHY instrument has had issues with degradation and calibration in the SWIR spectral range, and efforts to retrieve CO are hampered by the failure of a significant number of detector pixels and the growth of a layer of ice on the detector array (Gloudemans et al., 2005(Gloudemans et al., , 2008)).Borsdorff et al. (2016) discussed a recalibration of the measurements and presented a full-mission CO data set, which is restricted to clear-sky scenes over land.This SCIAMACHY CO data product for clear skies was validated with ground-based Fourier transform spectrometer (FTS) measurements provided by the Network for the Detection of Atmospheric Composition Change (NDACC) and the Total Carbon Column Observing Network (TC-CON) (de Laat et al., 2010;Borsdorff et al., 2016), and it was also validated with airborne measurements provided by the MOZAIC/IAGOS project (de Laat et al., 2012;Borsdorff et al., 2016).The successful validation of this recalibrated data set showed that SCIAMACHY observations reveal meaningful information about atmospheric CO over the full-mission period.
SCIAMACHY's SWIR measurements of clear skies over land have a good sensitivity for the vertical column density of CO.However, the clear-sky CO data product requires a strict cloud filter over land and the rejection of all observations over oceans due to the low reflectivity of the ocean surface in the SWIR.Hence, a retrieval that works for cloudy scenes is very desirable in order to extend spatial coverage beyond the small fraction of SCIAMACHY observations, which are clear skies over land.Gloudemans et al. (2009) and Buchwitz et al. (2006) demonstrated that cloud contaminated measurements can be useful for the CO retrieval, because the high reflectivity of clouds decreases retrieval noise.By using SCIAMACHY observations under cloudy conditions, Gloudemans et al. (2009) investigated the outflow of CO from Indonesia over the oceans.However, to properly interpret these measurements, one has to account for the shielding and scattering effect of clouds.For this purpose, Vidot et al. (2012) and Landgraf et al. (2016b) proposed the Shortwave Infrared CO Retrieval (SICOR) algorithm for the operational processing of the S5P mission.In addition to trace gas absorption, SICOR estimates two cloud parameters: cloud optical thickness and cloud height.These parameters are used in a two stream radiative transfer solver 2S-LINTRAN to account for changes to the light path due to light scattering by clouds and aerosols.For cloudy conditions, the CO error induced by this radiative transfer solver occasionally reaches several percent (Landgraf et al., 2016b), but becomes marginal for regional measurement ensembles.The SICOR data product includes the retrieved CO column, its noise estimate and the column averaging kernel.The averaging kernel reflects the vertical sensitivity of the retrieval to CO, and it depends on the cloud parameters that were estimated for the observed scene.
In this study, we apply the SICOR algorithm to SCIA-MACHY's 2.3 µm SWIR measurements.Doing this allows us to evaluate the SICOR algorithm's performance on real cloud-contaminated measurements and to expand upon the data set produced by Borsdorff et al. (2016).Whereas that CO data set was limited to clear-sky retrievals over land, here we provide SCIAMACHY CO data for cloudy scenes over the oceans and both clear and cloudy scenes over land.The new data product includes column averaging kernels for individual measurements, which helps to validate the data product with vertical concentration profiles of CO provided by the MOZAIC/IAGOS project.We also validate the data product with TCCON and NDACC ground-based FTS measurements.The paper is structured as follows.Section 2.1 introduces the inversion of vertical column densities by profile scaling, Sect.2.2 explains the retrieval set-up for the SCIA-MACHY measurements and Sect.2.3 analyses the quality of the retrieved cloud parameters.In Sect.3.1 we present the data validation using MOZAIC/IAGOS aircraft measurements while Sect.3.2 focuses on the validation with NDACC and TCCON measurements.A summary and conclusions are given in Sect. 4.

Inversion method
The inversion of CO vertical column densities from SCIA-MACHY's 2.3 µm reflectance spectra utilises the profilescaling approach.This approach has been applied to SCIA-MACHY data before by Gloudemans et al. (2008) and Borsdorff et al. (2016), and its regularisation type is discussed in detail by Borsdorff et al. (2014).Here, we summarise the aspects of the retrieval that are needed for the later discussion.

Profile-scaling approach
The inversion is based on the assumption that the measurement y meas can be described by the forward model F in the bounds of the measurement error e y , namely (1) Here, the vector b comprises forward model parameters, which are known a priori.The state vector x contains all parameters to be retrieved including the column density of CO and other trace gases.
The forward model F requires as input the vertical concentration profile of the atmospheric trace gases, which we obtain by scaling a reference profile ρ ref with the corresponding column density of the trace gas in x.The reference profile does not change during the retrieval and is normalised by its vertical column density c: (2) Here, T indicates the transposed vector C with all elements C i = 1.So Eq. ( 2) approximates the vertical integration assuming that the entries of the CO profiles are given in subcolumns.
Equation ( 1) is inverted with respect to state vector x with the solution x ret using the least squares fitting approach, where we apply the Gauss-Newton algorithm to account for the non-linearity of the forward model.The retrieved vertical column density c ret is an element of the solution vector x ret and describes an effective column density due to the regularisation inherent to the profile-scaling approach.In the linear approximation, the effective column and the true atmospheric abundance are related by the equation, where a col is the column averaging kernel, e c is the column retrieval error due to the measurement error e y and ρ true is the true trace gas profile.Borsdorff et al. (2014) presented a numerically efficient algorithm for calculating a col .The column averaging kernel represents a weighted integration of the true vertical profile, taking into account the particular retrieval sensitivity.The differences between the true column, c true = Cρ true and c eff = a col ρ true , cannot be inferred from the measurement and this is known as the null-space or smoothing error of the retrieval (Borsdorff et al., 2014;Rodgers, 2000), Hence, when considering c eff as an estimate of the true column, data interpretation does not rely on the use of the total column averaging kernel but the null-space error e null becomes part of the error budget.When c eff is seen as an effective column with its vertical sensitivity described by the column averaging kernel, data analysis focuses more on the information that can be inferred from the measurements and reduces the impact of the a priori choice of the reference profile.This requires a proper use of the total column averaging kernel as shown in Eq. ( 3) and the null-space error does not contribute to the error budget (see also Wassmann et al., 2015).
Finally, the measurement noise described by the measurement covariance matrix S y introduces noise on our retrieval product, which is characterised by the retrieval noise covariance matrix, ( Here G is the gain matrix of the inversion and its definition is given by Borsdorff et al. (2014).Therewith, we have defined all diagnostic tools for our retrieval.A detailed overview of the profile-scaling approach is given in Borsdorff et al. (2014).

Retrieval settings
For SCIAMACHY CO retrievals, we use the recalibrated SCIAMACHY spectra including estimates of the measurement noise as described by Borsdorff et al. (2016).The SICOR algorithm settings are very similar to those of the TROPOMI CO data processor (Landgraf et al., 2016b, a) but with adaptions to account for the instrument degradation of the SCIAMACHY instrument (Borsdorff et al., 2016).It comprises two main processing steps: first a nonscattering retrieval for cloud detection and subsequently a scattering retrieval to infer CO column abundances.For the SCIAMACHY measurements, we choose the spectral window 2311-2338 nm for both retrieval steps.This represents a wider retrieval window than in the TROPOMI settings, which are 2315-2324 nm for the non-scattering retrieval and 2324-2338 nm for the scattering retrieval (Landgraf et al., 2016b).This is necessary to account for the instrument degradation, e.g. the considerable loss of detector pixel by radiation damage and the light scattering of the ice-layer on the detector array of SCIAMACHY's channel 8 (Borsdorff et al., 2016).To account for radiometric errors of the SCIA-MACHY measurements, we apply a time-dependent radiometric correction derived from SCIAMACHY observations over the Sahara and a reliable model prediction of methane over this area.Moreover, the measurement noise is estimated from regular eclipse measurements and both aspects are described by Borsdorff et al. (2016).
The settings for the non-scattering retrieval are identical with the one described by Borsdorff et al. (2016) and in the first instance we retrieve the column density of CO, CH 4 , H 2 O and HDO without accounting for atmospheric scattering and clouds in particular.In that case, the difference T. Borsdorff et al.: SCIAMACHY CO retrievals for clear-sky and cloudy atmospheres between the retrieved CH 4 column and the a priori CH 4 column from the chemical transport model TM5 (Williams et al., 2013(Williams et al., , 2014) ) reflects the light-path effect of aerosols and clouds on the observations due to shielding and photon path enhancement by multi-scattering (Vidot et al., 2012).For the subsequent retrieval of CO, we use the retrieved nonscattering CH 4 column to provide a first estimate for the cloud height.For this purpose we integrate the TM5 vertical CH 4 profile from the cloud centre to the top of the model atmosphere and adjust the cloud height to match the retrieved non-scattering methane column.
The subsequent scattering retrieval is described in detail by Landgraf et al. (2016b).It estimates the trace gas columns of CO, H 2 O and its isotopologue HDO simultaneously with the cloud height h cld and cloud optical thickness τ cld , assuming a fully overcast scene.To estimate the cloud parameters from the SWIR measurements, the atmospheric CH 4 vertical profile concentration is fixed to accurate model estimates provided by the TM5 model.The scattering retrieval simulates a cloud as a horizontally homogeneous scattering layer with a triangular height profile in optical depth with a fixed half width of 1.5 km and infers the optical thickness and the cloud height from the methane absorption signal.In case of cloud heights < 3 km the cloud profile is cut off at the Earth's surface and is renormalised.The remaining a priori information and the molecular absorption spectroscopy are described by Borsdorff et al. (2016).

Evaluation of the retrieved cloud parameters
Compared to the previous work by Borsdorff et al. (2016), the new aspect of this study is the retrieval of CO column densities from cloudy SCIAMACHY measurements, from which we infer cloud parameters from the 2.3 µm methane absorption band.The effect of clouds on the retrieved CO column density is described with the column averaging kernels for each individual measurement.Obviously the retrieved cloud properties are effective parameters and so do not necessarily represent the real atmospheric situation, but describe clouds, cirrus and aerosols such that the simulated light path is sufficiently accurate to retrieve CO within the required accuracy.For the retrieval of effective cloud properties it is sufficient to describe the cloud by its optical depth and the asymmetry factor of the scattering phase function without analysing the underlying droplet size distribution and refractive index as discussed in detail by Landgraf et al. (2016b).In that sense the retrieved cloud parameters depend on the forward model of the retrieval algorithm and the spectral fit window, which makes their verification difficult.
To gain experience with the SICOR cloud parameters, we compared the retrieved cloud height h cld with the SCIA-MACHY cloud top height product, processed with the Fast Retrieval Scheme for Clouds from the Oxygen A Band (FRESCO) algorithm.In contrast to SICOR, this algorithm infers cloud top height and cloud fraction from the Oxygen A band around 760 nm assuming an elevated Lambertian reflector as a cloud model (Wang et al., 2008).This comparison is particularly worthwhile since SCIAMACHY's NIR measurements do not suffer from the formation of an ice layer on the detector module like the SWIR measurements (Gloudemans et al., 2008).Figure 1 shows a scatter plot of the SICOR and FRESCO cloud heights for 1 year of SCIAMACHY observations in year 2003 for an ocean region with a latitude range from 30 • S to 0 • and a longitude region from 30 • W to 0 • .We only consider measurements with a significant cloudiness indicated by SICOR cloud optical depth of τ cld > 1.The cloud parameters are highly correlated with a Pearson correlation coefficient r = 0.9.A linear regression provides a slope of 1.01 and an intercept of 300 m, indicating a valuable cloud product from the SWIR measurements of SCIA-MACHY.Even for later years of the mission we find a similar agreement (e.g.r = 0.8, slope = 0.9, intercept = 431 m for the year 2010).The remaining small difference between the retrieved cloud height and the FRESCO cloud top height may originate from the different cloud models used.For example, FRESCO simulates clouds as an elevated Lambertian surface, whereas SICOR fits the centre height of the triangular height profile.This comparison should not hide the fact that the validity of the SICOR effective cloud parameters can only be demonstrated by comparing the SCIAMACHY CO data product for cloudy observations with independent validation measurements, which is the subject of the next section.
In the following, we use the SICOR cloud parameters to classify the retrievals with respect to cloudiness.We distinguish between three CO retrieval conditions: (1) clearsky observations over land with τ cld < 0.3 and h cld < 1 km, (2) cloudy observations contaminated by optically thick low clouds with τ cld > 2 and h cld < 1.5 km and (3) cloudy observations contaminated by optically thick high clouds with τ cld > 2 and 1.5 km < h cld < 8 km.For these categories, Fig. 2 shows typical column averaging kernels selected from the SCIAMACHY CO retrievals over Australia for the year 2003.
Column averaging kernel values of one represent the ideal value for a vertical integration (see Eq. 2) and values < 1 indicate reduced retrieval sensitivity, e.g. because of atmospheric shielding by clouds and aerosols.Values > 1 are typical for a profile-scaling approach.In case of an optically thick cloud the sensitivity of the measurement for CO below the cloud is lost.The inversion scales the entire vertical profile of CO based on its sensitivity to CO above the cloud.Hence, the sensitivity of the retrieved CO column with respect to the CO concentration above the cloud is enhanced (> 1) and at the same time the sensitivity for CO below the cloud is reduced (< 1).These features are clearly reflected in Fig. 2 and discussed in more detail by Borsdorff et al. (2014).
So under clear-sky conditions we can retrieve CO vertical column densities with a sensitivity close to 1 at all altitudes (see yellow line in Fig. 2).Hence, the corresponding null-space error is small (Buchwitz et al., 2004;Gloudemans et al., 2008) and a direct comparison of the retrieved column with CO column reference measurements is possible while neglecting the effect of column averaging kernels (Borsdorff et al., 2016).For cloudy retrievals the comparison must account for averaging kernel effects.Clouds can easily lead to null-space errors > 30 % depending on the discrepancy between the true vertical trace gas profile and the reference profile to be scaled by the retrieval (Borsdorff et al., 2014).
The null-space does not lead to a problem for application like data assimilation if the averaging kernel is applied properly.Exploiting the altitude sensitivity given by the averaging kernel, the atmospheric state of the model can be adjusted adequately by the assimilation scheme.Also, for the validation of the CO data product, the null-space error does not impose principal limitation where Eq. ( 3) accounts for cloud effects in our retrieval when comparing satellite measurements with independent profile soundings.Wassmann et al. (2015) demonstrated this strategy for ozone column retrievals from GOME-2 measurements using ozonesonde measurements for validation.In practice, however, this approach is often hampered by the small number of available validation measurements.
CO retrievals under cloudy conditions represent an interesting addition to our clear-sky data product.Here, the retrieved column is mostly sensitive to CO above the cloud and for varying cloud height, and due to the shielding effect of clouds, different altitudes are probed.For the interpretation of errors, it is important to note that for the scaling of a reference profile, the interpretation of relative biases is the same for clear-sky and cloudy retrieval.In both cases, it indicates the relative error to the CO concentrations at all altitudes the retrieval is sensitive to.Furthermore, the radiometric precision of SWIR measurements for cloudy scenes is generally much better than for clear-sky observations because of the brightness of clouds.In the 2.3 µm spectral range this represents an important asset for data exploitation, particularly for SCIAMACHY, which suffers from a poor radiometric precision.Also for TROPOMI, which has a much better radiometric performance, CO retrievals over cloudy scenes represent an important addition for ocean scenes where, due to the low reflectivity of ocean water, no information about atmospheric CO can be inferred from clear-sky observations.
To illustrate this, Fig. 3 shows the signal-to-noise ratio (SNR) and CO retrieval noise as a function of the SICOR cloud optical thickness τ cld for an ocean region with a latitude range from 30 • S to 0 • and a longitude range from 30 • W to 0 • .For a cloudless scene τ cld → 0, the signalto-noise ratio is less than 25 and the CO retrieval error is very high.For the SICOR cloud optical thickness of 1-2 the SNR rises above 100 and the CO retrieval noise reduces to 5 × 10 17 molec cm −2 .For larger cloud optical thickness, the SNR and CO precision saturate around τ cld = 8 where the SNR reaches 250 and the corresponding CO retrieval noise is about 2 × 10 17 molec cm −2 .
Figure 4 shows that much more data become available when considering cloudy-sky retrievals under optically thick low and high cloud conditions in addition to clear-sky retrievals.Between January 2003 to the end of the SCIA-MACHY mission in April 2012, 70 % of all data are inferred from cloudy-sky observations (40 % over the oceans and 30 % over land).Over the oceans only cloudy-sky retrievals are possible but over land the fraction of clear-sky to cloudy-sky retrievals varies.For example, the data coverage over the Sahara region is dominated by clear-sky retrievals while the Amazon region and Indonesia show a majority of cloudy-sky retrievals.Averaged globally, 55 % of all measurements over land are cloudy-sky retrievals.Hence for SCIAMACHY, our CO retrieval algorithm for cloudy conditions permits the CO data product over the ocean, and in addition has the potential to improve the product over land.

Validation
The quality of our CO retrieval for cloudy atmospheres needs to be demonstrated through validation.First, we validate the SCIAMACHY CO retrievals with airborne profile measurements of the MOZAIC/IAGOS project, based on in situ measurements on commercial airliners (Nédélec et al., 2003).Following this, we complete our validation using CO groundbased measurements of the TCCON and NDACC networks.
To ensure the quality of the SCIAMACHY CO column retrievals, we perform an a posteriori quality filter on the individual CO retrievals: 1.The χ 2 values of the spectral fit must be < 40.
2. The mean signal-to-noise ratio of the measurements in the fit window must be > 20.
3. The noise of the retrieved CO, HDO and H 2 O column must be below an upper threshold, namely CO < 1×10 19 molec cm −2 , HDO < 1×10 20 molec cm −2 , and These criteria represent a weak data filtering to remove outliers due to unphysical retrievals.For example, the signalto-noise threshold corresponds to clear-sky measurements over the oceans (see Fig 3) for which a stable inversion is not possible.The retrieval would result in unphysical outliers and are thus rejected by the filter.Hence, by definition the data filter does cause some data gaps over the ocean.However, its effect on land observation is minor

Airborne CO profile measurements (MOZAIC/IAGOS)
The MOZAIC/IAGOS project provides profile measurements of reactive gases performed on board long-distance passenger airliners.Since 1994, in situ profile measurements are recorded during the ascent and descent phases of more than 40 000 flights.These observations are used to derive vertical column densities of CO with a precision of about 5 % (Nédélec et al., 2003).The recorded profiles do not represent a strict vertical intersection of the atmosphere but at lower altitudes atmospheric constituents are measured close to the airport, whereas at higher altitudes the distance to the airport may reach 200-400 km.De Laat et al. (2014) showed that in most cases the corresponding collocation errors with SCIA-MACHY observations imposed by the aircraft ascent and descent phases average out to a large extent when analysing temporal averages.
In this study, we consider the vertical profile measurements at three cities: Tehran and Beijing, which are known to be affected by strong local pollution events (de Laat et al., 2012), and Windhoek, where the atmospheric CO abundance is only slightly influenced by local emissions, resulting in spatially homogeneous CO fields.The Windhoek measurement site is situated in an area with higher surface albedo and thus we obtain high radiometric precision of SCIAMACHY clear-sky measurements.Table 1 provides more details on these sites.From the MOZAIC/IAGOS data set, we select only those profiles that reach at least an altitude of 9 km with data gaps < 1 km.The mean maximal altitude of the selected profiles is 11 km, which overall indicates a sufficient overlap between the aircraft measurements and the sensitivity of the retrieval with cloud heights < 8 km.Above the maximum flight altitude, the CO profiles are extended by simulated CO profiles of the TM5 chemical transport model.Furthermore, towards the ground level, we extend the profiles assuming a constant CO mixing ratio, which ensures the numerical stability of the extrapolation.When the CO column density is derived from the aircraft measurement in this manner its ac- Here, cloudy-sky retrievals comprise all data products of categories 2 and 3, defined in Sect.2.3.curacy is estimated to be a few percent (de Laat et al., 2012).
Numerical experiments using the lower data points of the profile to estimate the gradient in the CO concentration for linear extrapolation led to instabilities and thus this approach is not used here.To collocate SCIAMACHY retrievals with MOZAIC/IAGOS measurements, we select all SCIA-MACHY CO retrievals with a ground pixel in a radius of 850 km around the airport site.The temporal collocation criterium is chosen dynamically for each individual MOZAIC/IAGOS measurement and varies typically between 7 and 30 days.Centred around the recording time of the aircraft measurements, the temporal collocation window is chosen such that the average of all spatially collocated SCIAMACHY measurements have a precision < 10 17 molec cm −2 .Therewith, we ensure that the precision of the SCIAMACHY validation measurements is sufficient for our purpose.For the validation with MOZAIC/IAGOS observations, we consider SCIAMACHY measurements for clear-sky and cloudy conditions with high, optically thick clouds corresponding to condition (1) and (3) defined in Sect.2.3 and calculate the collocated MOZAIC/IAGOS CO column density from the profiles in two ways: first we perform the vertical integration of the MOZAIC/IAGOS profile after accounting for elevation differences between the SCIAMACHY ground scene and the airport site.Second, we apply the column averaging kernels of all collocated SCIAMACHY observations to the MOZAIC/IAGOS profile and calculate their average.Finally, we average the collocated SCIAMACHY CO column densities accordingly.The two approaches corresponds to the different interpretation of the SCIAMACHY CO product as discussed in Sect. 2. Figures 5 and 6 show corresponding time series for the three airport sites.Per site, we calculate the mean and standard deviation of the CO time series as a diagnostic tool, which is depicted in Fig. 7 for the clear-sky and cloudy SCIAMACHY data sets.The figure clearly shows that applying the SCIAMACHY column averaging kernels to the MOZAIC/IAGOS profiles improves the validation for all sites for both clear-sky and cloudy-sky conditions.For clear-sky measurements, this bias improvement is small and is about 9.5 ppb for Tehran, 4.0 ppb for Beijing and only 0.5 ppb for Windhoek, which confirms the findings by Borsdorff et al. (2014), Gloudemans et al. (2005), and Buchwitz et al. (2004).In case of clouds, we notice a significant improvement of the bias: 122.2 ppb for Tehran, 22.6 ppb for Beijing and 7.3 ppb for Windhoek.
De Laat et al. ( 2012) explained the larger errors for clearsky observations with strong local pollution at the airports, which affects the airborne measurements but are not seen by the satellite due to the coarse spatial resolution.This is concert with the extremely large values of the MOZAIC/IAGOS CO column densities time series in Fig. 5. Hence, we expect a better agreement when we consider SCIAMACHY retrievals for high clouds because of the atmospheric shielding of the spatial heterogeneity of CO.Figures 5 and 7 confirm this.Applying the column averaging kernels of the cloudy-sky retrievals to MOZAIC/IAGOS measurements significantly improves the comparison with SCIAMACHY retrievals, supports the error interpretation by (de Laat et al., 2012) and demonstrates the data quality of SCIAMACHY cloudy-sky retrievals.
Another interesting feature of our comparison is the better agreement for Beijing than for Tehran, where at the same time the SCIAMACHY mean CO column mixing ratio at Tehran is about 43.0 ppb smaller than for Beijing with about 16.0 ppb less scatter.This may hint at a larger representation error for Tehran than for Beijing, meaning that SCIA-MACHY, with its coarse spatial sampling, captures the enhanced CO concentrations over Beijing better than those over Tehran and so it explains the differences we see in Fig. 7 for these sites.
We complete our SCIAMACHY CO validation that is based on MOZAIC/IAGOS aircraft measurements by analysing observations from the airport at Windhoek.Here, the spatial CO distribution is less affected by local sources and so representation errors are less relevant.Moreover, the high surface albedo ensures clear-sky retrievals with high precision.We notice a small improvement when applying the SCIAMACHY cloudy-sky averaging kernels to the aircraft measurements.The mean and standard deviation in Fig. 7 show that SCIAMACHY cloudy-sky and clearsky retrievals agree well with the MOZAIC/IAGOS data for Windhoek.At this site the reference profile ρ ref from the TM5 model is more accurate and hence applying the averaging kernels to the MOZAIC/IAGOS validation measurements is less important.Consequently, the fact that clear-sky and cloudy retrievals result in the same bias indicates that the accuracy for CO is comparable at altitudes that the measurements are sensitive to.

Ground-based Fourier transform spectrometers (NDACC/TCCON)
The TCCON and NDACC networks perform direct sunlight measurements with ground-based Fourier transform spectrometers under clear-sky conditions.The Infrared Working Group (IRWG) performs measurements in the mid-infrared spectral range at 4.8 µm and is part of NDACC (http://www.ndsc.ncep.noaa.gov/).Thereby, it supplies CO vertical column densities that we transformed to column averaged mixing ratios by calculating the dry-air column from the surface pressure at the station sites.The TCCON instruments measure in the same spectral range as SCIAMACHY and provide the vertical column densities of trace gases including CO with high precision (Wunch et al., 2010(Wunch et al., , 2011)).
Because of the lack of profile information, it is not possible to apply averaging kernels, and NDACC/TCCON data must be compared directly with the SCIAMACHY cloudysky retrievals.The comparison depends critically on the CO a priori profile information, and so we select near-coast measurement sites in remote areas, where the TM5 model predic-  tion of the relative CO profile is most reliable: Ny-Ålesund, Wollongong (Griffith et al., 2014), Lauder (Sherlock et al., 2014), Mauna Loa and Reunion.More details about the stations are given in Table 2. Figures 8 and 9 show time series of 30-day median values derived from the ground-based FTS measurements and the collocated SCIAMACHY retrievals.Using the same criteria (i.e collocation radius and quality filtering) as in Sect.3.1, missing values of the ground-based FTS time series were interpolated (open circles) using the TM5 model as described by Borsdorff et al. (2016).
The upper panel of Fig. 8 illustrates that data validation using ground-based measurements on small islands like Mauna Loa and Reunion are challenging for the clear-sky SCIA-MACHY CO retrievals over land.Only a few satellite land observations can be collocated with the ground-based measurements.Moreover, these measurements have low surface albedo, which results in a large retrieval noise > 100.0 ppb for single soundings.So for these locations, only a few 30day median values can be determined, which are still dominated by the retrieval noise as shown in the left panel of Fig. 10.The scatter of individual SCIAMACHY retrievals is described by the half difference of the 15.9th and the 84.1th percentile S and is taken to be an analogue for the standard deviation of a normal distribution.Further, N is the mean retrieval noise of the single measurements of the monthly medians.The correlations between the FTS and clear-sky SCIA-MACHY measurements over land are poor (about r = 0.3) and the high standard error (> 17.0 ppb) indicates that under these conditions even the overall mean value is difficult to estimate.The situation improves for measurement sites on larger islands like Lauder and Ny-Ålesund.We found more collocations between SCIAMACHY land observations and FTS measurements (Fig. 9) which led to a smaller standard error of the mean (< 8.0 ppb), and consequently a more reliable estimation of the overall mean.However, the 30-day median values are still dominated by the retrieval noise due to the low surface albedo of the observed scenes, and so the correlations between the FTS and SCIAMACHY measurements are still low (about r = 0.3, see Fig. 10).For SCIA-MACHY, clear-sky measurements over land, we achieved the best results for FTS measurements at Wollongong with high surface albedo.The upper panel of Fig. 9 shows that in this case the 30-day CO medians agree well with the FTS measurements.Moreover, the global mean can be estimated with a small standard error, and the Pearson correlation coefficient for FTS and SCIAMACHY measurements is r = 0.5, indicating that the CO seasonality of the FTS measurements can be captured reasonably well by the SCIAMACHY observations.
When analysing SCIAMACHY cloudy-sky CO retrievals with h cld < 1.5 km and τ cld > 2 corresponding to condition (2) defined in Sect.2.3, the comparison improves for all stations (see lower panel of Figs. 8 and 9).Due to the high reflectivity of the clouds, the single measurement retrieval noise is reduced to < 30.0 ppb and many more ground pixels can be collocated with SCIAMACHY.Figure 10 shows that the standard error of the mean bias falls below 2.0 ppb.For Lauder, the mean bias improves from b = −12.0 to b = −2.0ppb but stays nearly the same for Ny-Ålesund.In these cases, the correlation increases to r = 0.6 and 0.7 respectively.For Wollongong, the mean bias for clear-sky and cloudy-sky retrievals is comparable (about 4.0 ppb), and the correlation improves from about r = 0.5 to 0.8 for the cloudy retrievals.Here the majority of the cloudy measurements are observations over the oceans and so these results indicate that there is no evidence for a land-sea bias of the SCIAMACHY CO retrieval because, considering all reported cloudy-sky re- trievals, we estimate a mean bias b = −6.0ppb.This may be due to remaining effects of the retrieval noise or due to the loose collocation criteria.Also, issues due to erroneous radiometric calibration of SCIAMACHY SWIR measurements cannot be excluded.
The presented validation analysis provided sufficient confidence to reprocess the global full-mission SCIAMACHY CO data set including clear-sky and cloudy-sky observations.In particular, the validation with NDACC and TCCON measurements revealed that the CO column, retrieved from measurements with low, optically thick clouds, can be used as an estimate of the true column (i.e.ignoring effects of the column averaging kernel) in the absence of local sources of CO.The full-mission averaged CO product is shown in Fig. 11 and is based on clear-sky and cloudy observation contaminated by optical thick low clouds (h cld < 1.5 km).This figure indicates that the cloudy-sky ocean retrieval adds valuable information to the clear-sky data product.For example, over the Atlantic Ocean the CO outflow due to biomass burning in central Africa becomes clearly visible as well as the CO transport over the Pacific Ocean due to pollution from China and the wildfires in Indonesia.All this reveals information about the global transport of pollution in the atmosphere.Future research must demonstrate the merit of this observations for studying global transport of pollution in the Earth's atmosphere.

Conclusions
In this study, we derived a full-mission SCIAMACHY CO column data set that comprises retrievals from clear-sky and cloudy-sky reflectance measurements in the spectral range 2311-2338 nm over land and ocean scenes.The inversion uses the SICOR CO retrieval code that is developed for the operational data processing of the Sentinel-5 Precursor mission.It allows us to retrieve effective cloud parameters simultaneously with trace gas columns.The data product includes the CO column and its column averaging kernel for each individual sounding, and so it provides information on the vertical CO retrieval sensitivity, which changes with the cloudiness of the observed scene.This study focused on the validation of SCIAMACHY CO retrievals for cloudy-sky observations with MOZAIC/IAGOS aircraft measurements and TCCON and NDACC ground-based observations.It represents an significant extension with respect to the previous work by Borsdorff et al. (2016), presenting a similar data set for clear-sky scenes over land.
The effective cloud parameters (cloud height h cld , cloud optical thickness τ cld ) are needed to properly simulate the atmospheric light path as part of the CO retrieval.Generally, the validation of effective cloud parameters is difficult, and in this study the cloud parameters are validated implicitly by the validation of the CO data product from cloudy-sky SCIAMACHY observations with independent atmospheric measurements.To gain confidence in the cloud product, we compared the cloud height of the SICOR data product from one year of SCIAMACHY observations over the ocean with the corresponding cloud height of the FRESCO data product, which is inferred from SCIAMACHY measurements of the O 2 A band.We found a strong correlation (r = 0.9) between both products and an overall offset of the SICOR cloud height versus FRESCO of 300 m.Even for later years of the mission, e.g.2010 we find a similar agreement (r = 0.8, slope = 0.9, intercept = 431 m).This agreement shows that the cloud height retrievals from the different spectral ranges of SCIAMACHY are consistent, even though the SWIR measurements are affected by radiometric degradation of the instrument.
SICOR uses the profile-scaling approach to infer CO vertical column densities from the measurement, which involves a regularisation of the inversion problem.When interpreting the retrieved column as an estimate of the truth, a null-space error is introduced.This null-space error depends on the accuracy of the profile shape to be scaled by the inversion.Generally for clear-sky observations the null-space error is small, but for cloudy conditions it can easily exceed 30 %.Here, clouds shield the atmosphere below and the a priori CO profile shape is used to add the lacking information.The sensitivity of the retrieved CO column with respect to the true CO density is provided by the column averaging kernel for each individual retrieval.For the validation, we can thus interpret the retrieved column as a vertically integrated CO column density weighted by the column averaging kernel, and so the null-space error becomes less relevant for the comparison of the retrieval product with independent atmospheric measurements of the CO profile.
Validating SCIAMACHY retrievals with MOZAIC/IAGOS airborne measurements confirmed the approach.Direct comparison of the MOZAIC/IAGOS CO columns estimated at Beijing, Tehran and Windhoek with collocated SCIAMACHY cloudy-sky retrievals of the CO column show a small bias of 0.5-9.5 ppb.However, for clear-sky SCIAMACHY observations, the bias exceeds 120.0 ppb for Tehran and 30.0 ppb for Beijing which is in agreement with previous studies (Borsdorff et al., 2016;de Laat et al., 2012).This indicates the large relevance of the null-space error for the comparison of cloudy observations with other data.The difference became much smaller when we accounted for the retrieval sensitivity in the comparison by applying the column averaging kernels to the airborne measurements.Here the bias reduced to 50.0 ppb for Tehran and 8.0 ppb for Beijing.We attribute the remaining error for Tehran to the different sampling of different air masses by the satellite and aircraft measurements.For Beijing, which has a much more extended CO surface emission spatially, these errors are less relevant.For Windhoek, the comparison of aircraft and satellite observations shows differences of 0.2 ppb when the averaging kernel is applied and 8.0 ppb otherwise.For this site, the vertical shape of the a priori CO profile is well known and so it explains the smaller bias for this site.
We completed our validation study using ground-based FTS measurements from the NDACC and TCCON networks for the coastal sites Ny-Ålesund, Lauder, Mauna Loa, Reunion and Wollongong.At these validation sites, an independent measurement of the CO profile is not available, and so we assumed the shape of the CO profile.Hence, we directly compare the column densities from FTS with those of SCIAMACHY measurements for both clear-sky conditions and cloudy conditions with low clouds.Considering only clear-sky SCIAMACHY observations over land at Lauder, Mauna Loa and Reunion, the comparison was dominated by the SCIAMACHY retrieval noise because of the low surface albedo and the insufficient number of collocations.Filtering the SCIAMACHY data set to select for optically thick low clouds over land and oceans, we found good agreement with the FTS ground-based measurements with a correlation coefficient of r = 0.6-0.7.For Wollongong, the high surface albedo led to CO retrievals from SCIAMACHY clearsky measurements that compare well with FTS ground-based measurements with a mean bias of b = −4.3ppb and a correlation of r = 0.5.Even for this site, the cloudy retrievals improved the comparison with a high correlation of r = 0.8.The mean bias stayed the same within its uncertainty, which demonstrated the overall consistency of the clear-sky and cloudy-sky data product.
Finally we processed the full-mission data record of SCIA-MACHY SWIR measurements.This data set clearly demonstrates the asset of cloudy-sky CO retrievals over oceans, providing a global coverage of the SCIAMACHY CO data product.The correct interpretation of these data requires the use of the column averaging kernel.In future, the data may be used to discriminate the vertical distribution of CO in the atmosphere, e.g. by means of data assimilation.For this, retrievals under optically thick clouds with varying cloud heights are essential because the shielding of the atmosphere below the cloud reveals information about CO in different altitudes.
It is the first time that the operational TROPOMI CO algorithm is tested successfully on real data for clear-sky and cloudy atmospheres, which is an important milestone for the preparation of the S5P mission.Although TROPOMI SWIR measurements will have the same spectral coverage and resolution as SCIAMACHY, we expect a much better data product due to its better radiometric performance and its better spatial resolution and sampling.After the launch of SP5, this techniques used in this study will allow us to use IA-GOS/MOZAIC and FTS measurements from the TCCON and NDACC networks for the validation of TROPOMI CO.Additionally, using the same retrieval approach for SCIA-MACHY and TROPOMI CO retrievals will help to provide a consistent long-term CO data set for both missions.

Figure 1 .
Figure 1.Scatter plot of the retrieved cloud height h SICOR in kilometres against that provided by Fresco h FRESCO .Data are shown for the year 2003 over the ocean in the latitude/longitude box [(30 • S, 30 • W), (0 • ,0 • )] with cloud optical thickness > 1.

Figure 2 .
Figure 2. Example SCIAMACHY CO column averaging kernels over Australia in 2003 as a function of altitude for the following cases: clear-sky (yellow line), a cloud with 0.7 km centre height (dashed line) and a cloud with 2 km centre height (solid line).Cloud optical thickness of 2.5 for the relevant cases.

Figure 3 .
Figure 3. (a) SNR in the SCIAMACHY CO retrieval window.(b) CO retrieval error CO (as a function of the retrieved cloud optical thickness τ cld ).Data are shown for the year 2003 for a latitude/longitude box [(30 • S, 30 • W), (0 • , 0 • )] over the ocean

Figure 4 .
Figure 4. Percentage of cloudy-sky retrievals of all retrievals from January 2003 to the end of the SCIAMACHY mission in April 2012.Here, cloudy-sky retrievals comprise all data products of categories 2 and 3, defined in Sect.2.3.
shows good agreement between SCIAMACHY and MOZAIC/IAGOS measurements with clear seasonality.

Figure 7 .
Figure 7. Mean CO column mixing ratio (a, b) and standard deviation (c, d) of the time series shown in Figs. 5 and 6, SCIAMACHY (black), MOZAIC/IAGOS (yellow) and MOZAIC/IAGOS with SCIAMACHY column averaging kernels applied (pink).(a, c) Clear-sky SCIAMACHY retrievals and (b, d) SCIAMACHY retrieval for optically thick high cloud conditions.

Figure 8 .
Figure 8. 30-day median of CO column averaged mixing ratios measured by SCIAMACHY (black) and at two NDACC-IRWG stations (pink).(a) SCIAMACHY clear-sky retrievals, (b) SCIA-MACHY cloudy-sky retrievals with optical thick low clouds.Open circles denote interpolated values for periods in which no NDACC-IRWG measurements are available.The scatter S and the mean retrieval noise N for individual retrievals in the monthly bins are shown.

Figure 9 .
Figure 9. Same as Fig. 8, for TCCON measurements indicated in blue.For Wollongong and Lauder the GGG2014 release of TCCON was used.Except for Ny-Ålesund data, release GGG2012 is used because the newer release GGG2014 is not yet available for this site.

Figure 10 .
Figure 10.Mean bias SCIAMACHY -FTS (a, b), standard error of the mean bias (c, d) and Pearson correlation coefficient of SCIA-MACHY with FTS (e, f) derived from monthly medians as shown in Figs. 8 and 9. Measurements of NDACC (a, c, e) and TCCON stations (b, d, f) are compared with clear-sky (grey) and cloudy (coloured) SCIAMACHY retrievals.

Figure 11 .
Figure 11.CO column averaged mixing ratios in ppb over land and ocean from clear-sky and cloudy-sky measurements for optically thick low cloud conditions.The values are averaged from January 2003 to the end of the SCIAMACHY mission in April 2012.

Table 1 .
MOZAIC/IAGOS airports used for validation.The temporal coverage with the SCIAMACHY mission is given in years.

Table 2 .
Same as Table 1, but for NDACC and TCCON sites.