2.1 Versions of Level-1 data

The uncalibrated L0 data are processed into two sublevels of L1: levels 1b and 1c. The SCIAMACHY specific calibrations applied to the L0 data are described in Slijkhuis et al. (2001). The two sublevels differ in the application of calibration steps: L1c data are processed by full calibration and contains geolocated, spectral radiance and irradiance products. Specifically the calibrated L1c data are produced using ESA's SciaL1C program of v3.2.6. We make use of three different versions of L1c (hereafter called L1) data products described below:

Table 2Parameter settings in the OPERA algorithm for SCIAMACHY nadir O₃ profile retrieval.

Download Print Version | Download XLSX

1. v7. This is SCIAMACHY L1 version 7.04−W released in early 2012, where “W” means the processing stage flag and 7.04 is the software version number used for converting L0 into L1 data.

2. v7_mfac. This version is identical to the one above, except here we use the degradation correction factors, so-called “m-factors”, that were provided independently as auxiliary data files¹. The data structure allows us to turn on and turn off the degradation correction independent of other calibrations. The m-factors are determined by the monitoring of the light path, which is given by the ratio of the measured spectrum of a constant source (Sun) to that obtained for the same optical path at a given time. This gives therefore the degradation of the optical path as the instrument ages. These m-factors are simple multiplication factors to the solar spectrum after the absolute radiometric calibration (Gottwald and Bovensmann, 2011).

3. v8. This is version v8.02, which is the 2016 version of the SCIAMACHY L1 product. The main difference between this version and the one above is the implicit implementation of a standard degradation correction with scan-angle dependence. Specifically the radiometric calibration uses a scan mirror model which takes into account the physical effect of the contamination layers on the mirror. The degradation using this model gives a scan-angle dependence (Bramstedt, 2014).

2.2 OPERA retrieval algorithm

The OPERA, developed in KNMI (van Oss and Spurr, 2002; van der A et al., 2002), retrieves the vertical ozone profile using nadir satellite observations of scattered UV light from the atmosphere. The algorithm makes use of radiative transfer computations of the top-of-atmosphere radiances given the atmospheric scattering and absorption parameters. The ozone absorption cross section decreases from 265 to 330 nm, which allows us to retrieve the amount of ozone as a function of atmospheric height.

In OPERA, the retrieval of ozone profiles uses the maximum a posteriori approach Rodgers (2000). The state of the atmosphere and the measurement are given by the vectors x and y, respectively, and the two vectors are related by the forward model F, according to y=F(x). The solution is given by the following three equations:

where $\widehat{\mathit{x}}$ is the retrieved state vector, x_a is the a priori, A is the averaging kernel, x_t is the “true” state of the atmosphere, $\widehat{\mathbf{S}}$ is the retrieved covariance matrix, I is the identity matrix, S_a is the a priori covariance matrix, K is the weighting function matrix or Jacobian (which gives the sensitivity of the forward model to the state vector) and S_ϵ is the measurement covariance matrix.

The matrices A and $\widehat{\mathbf{S}}$ provide valuable information on the retrieval. The sum of the diagonal elements of A (i.e. the trace) is called the degrees of freedom for signal (DFS). The higher the DFS, the more the retrieval has learned from the measurement. However, with a low DFS most information in the retrieval is coming from the a priori. The total DFS gives the number of independent pieces of information present in the retrieval. The rows of A are called the smoothing functions, since they give an indication of how x_t is smoothed out over the layers of the retrieval. The diagonal elements of the covariance matrix give the variance at the corresponding altitude, while the off-diagonal elements are related to the correlation between the layers in the retrieval. For a comprehensive algorithm overview and retrieval configuration, along with application of the algorithm to GOME-2 data, we refer to Mijling et al. (2010) and van Peet et al. (2014).

The OPERA configuration chosen for application to SCIAMACHY data is given in Table 2. The ozone profile retrieval grid is chosen according to the Nyquist criterion. For SCIAMACHY data we find that setting the retrieval grid to 31 layers or more gives the same value for the DFS. Following van Peet et al. (2014), we have used the McPeters et al. (2007) a priori ozone profiles, because it is a recent climatology (includes ozone depletion) and, unlike other climatologies, it does not require additional parameters like total ozone. Since the McPeters et al. (2007) a priori profiles do not contain covariance matrices, these are assumed to have an exponential decrease in pressure and are scaled with a priori errors (for details see van Peet et al., 2014). OPERA can also use the Fortuin and Kelder (1998) climatology. The reflectance measurement errors are provided in the SCIAMACHY L1 data; a systematic relative error in the measured reflectance (“noise floor”) of 0.015 is added to the measurement errors. The noise floor is assumed the same for all L1 data.

In practice, the measured reflectance spectrum R_meas(λ) (see Eq. 1) is prepared in the beginning of the OPERA algorithm, which is then passed to the forward model. This model contains vertical atmospheric profiles, like temperature and a priori ozone profile, geolocation, cloud data and surface characteristics. The forward model computes Sun-normalised radiances at the top of the atmosphere at wavelengths determined from the measured instrumental spectral data. After multiplication with a high-resolution solar irradiance reference spectrum and convolution of simulated radiances and irradiances with the instrumental SF, the simulated reflectance spectrum R_sim(λ) is obtained. The inversion step that follows is based on the optimal estimation (OE) method requiring measurement, simulation and measurement errors in vector and matrix forms. An inversion using derivatives of simulated reflectances with respect to the desired parameter to be solved is carried out until convergence is reached or until the maximum number of iterations is reached. For a comprehensive description of the implementation, configuration and the model parameters of the algorithm, we refer to the OPERA manual (Tuinder et al., 2014).

Table 3Statistics of the uncertainties of the L2 ozone profile products that are retrieved using three different versions of L1 data sets (described in Sect. 2.1). This table contains the statistics of the curves shown in Fig. 3.

Column 1: L1 data year. Column 2: data set version. Column 3: number of states used. Column 4: median number of iterations ± standard deviation. Column 5: median degrees-of-freedom ± standard deviation. Column 6: median relative uncertainty of converged reflectance spectra in the wavelength band used. Column 7: standard deviation of quantity in Column 6. Column 8: relative uncertainty in retrieved ozone profile. Column 9: standard deviation of quantity in Column 8.

Download Print Version | Download XLSX

As mentioned above, the averaging kernel (AK) of a retrieval represents the measurement sensitivity with respect to the true state of the atmosphere Rodgers (2000). In Fig. 2 we show an example of the AK for an individual OPERA ozone profile retrieval for a SCIAMACHY state on 7 January 2004. The AK rows represent the smoothing of the true profile as a function of the ozone retrieval layers. These smoothing functions should peak at the corresponding retrieval level and the half-width is a measure for the vertical resolution of the retrieval at that altitude. For an ideal retrieval, the curve of each row will peak at the nominal layer height with a spread that gives the vertical resolution of the retrieval. It is clear from Fig. 2 that information on the ozone amount in a certain layer is also coming from other heights.

Figure 2Example of the averaging kernel shape for a subset of seven layers from a 31-layer SCIAMACHY ozone profile retrieval. The triangles are the pressure levels in hPa of the retrieval layers. This subset of seven retrieval layers was selected uniformly ranging from top to bottom for clarity.

Download

4.1 Ozone profile validation comparison between versions v7, v7_mfac and v8 for 2003 and 2009

In Fig. 7 we show the ozone profile validation of the three data sets for two years, 2003 and 2009. The results are given for five latitude bands: 90 to 60^∘ N, 60 to 30^∘ N, 30^∘ N to 30^∘ S, 30 to 60^∘ S and 60 to 90^∘ S. For reference, the uncertainties (25–75 % percentiles) in the relative differences are shown only for v8 for all the latitude bands and both years as shaded regions. Each curve is the median difference between retrieved ozone column per layer and the averaging-kernel smoothed ozone sonde value, normalised by the ozone sonde profiles. In the lower-middle stratospheric region a better agreement of the retrieval with the sonde is expected. Qualitatively, it seems that for most latitude regions ozone profiles retrieved from L1 v8 compare better with ozone sondes than profiles retrieved from the earlier L1 versions. For v8 profiles the comparison is generally better in 2003 than in 2009. The quantitative comparison results are given in the following.

The relative differences between the retrieved profiles and the sonde profiles are given in Table 4, where the top half of the table shows the results for the years 2003 and for data sets v7, v7_mfac and v8, for three zones: Southern Hemisphere (SH) from 90 to 60^∘ S, tropics (Tr) from 30^∘ S to 30^∘ N, and Northern Hemisphere (NH) from 30 to 90^∘ N. In the bottom half of the table these quantities are given for the year 2009. From Table 4 we see that the retrieved profile deviations (median differences in percent) in the stratosphere are systematically smaller than in the troposphere. These large spreads in deviations of satellite retrievals from sondes are also visible in Fig. 7. Any deviation above ∼15 % in the stratosphere and above 20 % in the troposphere are shown in bold numbers for reference, as these are the required accuracy levels in the ESA Ozone CCI project (http://www.esa-ozone-cci.org/). This behaviour is probably not only due to the relatively worsening sensitivity of UV instruments in the troposphere. Rather, it might be due to the limited quality of the SCIAMACHY L1 data. Deviations in retrieved ozone profiles in the upper troposphere and lower stratosphere have been reported due to the ozone variability in a previous study of GOME-2 data (Cai et al., 2012). However, in the stratosphere the median deviations for 2003 are smaller for v8 for Tr and NH compared to the older data set versions, whereas for the SH the three different data sets give comparable deviations. The deviations for 2009 in v8 in the stratosphere are smaller for SH and NH than in the older data sets; the deviations are comparable in the Tr zone between all data sets. Comparison of the ozone profile deviations between 2003 and 2009 show larger values for 2009, suggesting that the quality of L1 v8 data has degraded.

4.2 Validation of v8 ozone profiles for 2003–2011

After establishing the differences between the quality of the three L1 data sets above based on the ozone sondes, here we show the validation results spanning the entire SCIAMACHY observation period using the latest L1 data set. In Fig. 8 we show validation results for the entire data set based on L1 v8 from the years 2003 to 2011, for the same five latitude zones as in Fig. 7. The left column shows results for the earlier years 2003–2006 and the right column for the later years 2007–2011. The solid lines correspond to the difference between satellite and sonde (convolved with satellite AK) normalised by the sonde profile. The number of collocated pixels for all the latitude bands for each year are listed in Table 5 along with median DFS, median number of iterations required to achieve convergence, the median solar zenith angle and the median deviations in percent for each latitude band in the stratosphere and troposphere. From Fig. 8 we note that validation for all latitude bands show in general smaller deviations from the zero line for earlier years than for later years. For later years the agreement with sondes becomes worse especially for tropics and NH bands. The median deviations given in Table 5 show often values higher than 20 % (in boldface) for the troposphere whereas these values are less than 15 % for the stratosphere, deeming them within specifications according to the Ozone CCI requirements (see Sect. 4.1). It should be noted, however, that the large deviations in tropospheric ozone are systematically higher than those for the stratosphere for all zones and the years even for v8, whereas such deviations are not observed for instance with GOME-2 nadir profiles (van Peet et al., 2014).

It can be seen from Fig. 8 that the vertical pattern of differences has changed significantly (and not just the absolute differences) over the instrument lifetime. This is important to consider when applying the profiles for ozone layer studies. This validation of ozone profiles based on L1 v8 suggests that the quality of nadir SCIAMACHY L1 data can still be improved upon. It affects the lowest troposphere in the beginning of the mission and gets worse due to instrument degradation. This is still uncorrected in the UV wavelength range. Please also note the higher deviations in the year 2003 in NH and Tr latitude zones in Fig. 8 as compared to other early years. This is a unique behaviour for 2003; the cause is not clear, but it is probably related to calibration or instrument changes early in the mission. Further improvements are needed (see Sect. 5).

Table 5Validation statistics of retrieved ozone profiles from L1 v8 for the period 2003–2011. Bold numbers in the medians of relative differences are exceeding ESA CCI accuracy thresholds.

Column 1: Year. Column 2: number of states per latitude zone: Southern Hemisphere (SH), tropics (Tr) and Northern Hemisphere (NH). Column 3: median DFS. Column 4: median number of iterations. Column 5: median solar zenith angle ± standard deviation for all states in that row. Column 6: medians of relative difference between retrieved and sonde profiles for stratosphere for SH, Tr and NH over the specified altitude range and for the corresponding latitude band. Column 7: same as in Column 6 for troposphere.

Download Print Version | Download XLSX

The deviations found in SCIAMACHY v8 ozone profiles can be compared, for instance, with GOME-2 ozone profile validation by van Peet et al. (2014). Their validation of tropospheric ozone showed deviations ranging from a few percent to ∼ −30 % for the Northern Hemisphere, whereas we find deviations that range from ∼ −10 to −45 % for the year 2008 (blue line in right panels Fig. 8). In the Southern Hemisphere, however, we find the deviations for year 2008 to range from a few percent to ∼ 20 %, which is more comparable to the range of a few percent to $\sim -\mathrm{15}\mathit{\%}$ found by van Peet et al. (2014).

5.1 In-flight slit function calibration using solar measurements

One of the spectral calibrations often performed to assess the behaviour of an instrument in flight is a fit of the instrumental SF (also called instrumental spectral response function). The SF parameters of SCIAMACHY were measured pre-flight and are provided as key data in the L1 product. The slit function S of the instrument has different functional forms depending on the channel. For channels 1–2 (relevant for our ozone profile retrieval) S is described by a single hyperbolic function:

The L1 key data provide the full width half maximum (FWHM), which is related to the a parameter in the equation above. This parameter can be solved in terms of the FWHM by using the fact that at the central point (λ₀) we have $S\left({\mathit{\lambda }}_{\mathrm{0}}\right)=\mathrm{1}/a^{\mathrm{2}}$. Thus, $a={\frac{\mathrm{FWHM}}{\mathrm{2}}}^{\mathrm{2}}$. At each given solar spectrum wavelength, λ_i, this functional shape is numerically computed between [−1,1] nm centred at λ_i. This shape can be manipulated with the following four parameters: additive constant (offset, O), multiplication factor (gain, G), displacement of the spectral peak (shift, D) and expansion and contraction of the spectral peak (squeeze, S). The high-resolution solar reference spectrum from Dobber et al. (2008) is modified with these four parameters to best match the SCIAMACHY measured solar irradiance (E(λ)). First the spectral parameters, shift D and squeeze S, are applied to Eq. (5):

followed by the radiometric parameters, gain G and offset O:

The unit of D is nanometre whereas O, G and S are numeric factors. All four parameters depend on wavelength. The parameters FWHM and λ₀ at each wavelength, taken from the v8 L1 key data, were identical for all the solar spectra throughout the mission. These values are given at certain wavelengths spread throughout channels 1 and 2 and were interpolated for the wavelengths in-between. An OE algorithm was used to solve for the best parameter values fitting the solar spectrum measurements in flight. The retrieved values of the four parameters at the solar spectrum wavelengths were then also applied to the Earth radiance spectra I(λ) interpolated at the solar spectrum wavelengths.

Thus for each wavelength of the solar spectrum the parameter values are computed using the above SF model. Each spectral peak of the solar reference spectrum can be transformed by manipulating the SF using offset, gain, shift and squeeze until it fits the measured spectrum. These spectral manipulations can be modelled as a polynomial of order n at the desired channels. The fit is checked by evaluating the relative difference between the solar irradiance measurement and simulation, which is the relative residual. The relative residuals for the best fit are shown in Fig. 9. Each curve is a residual for one solar spectrum, where the blue line is achieved by using only gain and offset in the model and the red line is achieved by including the shift and squeeze.

We find that the wavelength range 265–308 nm is the optimal range for obtaining the smallest residuals in Channel 1. However, the ozone profile retrieval algorithm requires spectra from 260 nm onwards. Because the part of Channel 1 above 308 nm is known to suffer of calibration problems, the SF fit in Channel 1 is performed for the range 260–308 nm. We ran the OE on all solar measurements of the SCIAMACHY mission for each day, which amounts to 3463 solar spectra. We convolved all spectra with {offset, gain, shift, squeeze} of polynomial order of = $\mathit{\{}\mathrm{2},\mathrm{15},\mathrm{2},-\mathit{\}}$ and found that the cost function for the majority of cases reaches less than 1 (as expected) within 10 iterations in the OE routine. By using a spectral shift in the range 265–308 nm instead of in the entire band 265–314 nm the residuals were significantly reduced (see left panel in Fig. 9). No SF fit was possible between 308 and 314 nm. Thus, from 308 nm onwards we used the Channel 2 spectra.

The residuals in Channel 2 are somewhat larger than those in Channel 1 (see Fig. 9). The dashed lines in both panels of the figure mark the ±2 % residuals for reference. Dividing the relevant range of 308–330 nm into smaller ranges to get smaller residuals did not reduce the residuals any further. For the optimal wavelength divisions, we found the smallest residuals given by the polynomial orders of the set of {offset, gain, shift, squeeze} = $\mathit{\{}\mathrm{1},\mathrm{4},\mathrm{1},\mathrm{2}\mathit{\}}$. In Channel 2, we find that using spectral shift and squeeze in the range 308–330 nm reduces the relative residuals significantly. The relative errors of the L1 solar irradiance in the wavelength range used for ozone profile retrieval, 265–330 nm, are only on the order of 10⁻⁵. However, the relative residuals are on the order of a few percent. So we expect this error to propagate into the ozone profile retrieval. There are strong residuals at around 279, 280 and 285 nm due to the MgI and MgII lines in the solar spectra. These spectral windows were therefore not used in the ozone profile retrieval (see Sect. 2).

Figure 10 shows the temporal behaviour of the SF parameters for Channel 1 (Fig. 10a) and Channel 2 (Fig. 10b) as density plots; the fitted parameter value for each wavelength and each day of the year is shown. The seasonal dependence of solar irradiance is observed, as expected, in the gain parameter for both channels. The other parameters do not show a clear seasonal dependence or a clear trend over the mission time.

5.2 Radiometric bias correction

Figure 11Mean ratio of observed (R_meas) to simulated (R_sim) reflectance spectra for 1 day for each year. The varying rainbow spectrum from blue to red colour labels the ratios for years 2003 to 2011. A horizontal line where the ratio is one is shown for reference.

Download

As shown above, spectral corrections like shift and squeeze at UV wavelengths can further improve the solar spectra by several percent (see Fig. 9). However, the expected improvement from this is less than the degradation and other remaining potential biases for the L1 data, which are increasing with the mission time. A preliminary analysis of this is shown in Fig. 11, where the mean ratio of observed to simulated reflectance spectra (R_meas∕R_sim) is plotted for the day of 24 June for years 2003–2011. There is a strong deviation of this ratio from one below 300 nm and this becomes worse for later years. A detailed study of this radiometric bias is beyond the scope of this paper. Please note the odd behaviour of the reflectance ratio around 283 nm. This was found to be due to a jump in the value of the Earth radiance at the transition of cluster 3 to cluster 4 owing to the difference in IT. This region is therefore blocked in our retrieval algorithm.

A L1 reflectance bias correction can significantly improve the quality of L1 data and will influence the validation and other results in this paper from their ozone retrievals. Furthermore, applying these bias corrections would also make it meaningful to apply the spectral SF corrections that can potentially improve the ozone retrievals further. Thus a detailed study of the effect of such bias corrections on the L1 data can be investigated against the quality of the ozone retrieval algorithm to better understand the quality of the SCIAMACHY nadir data.