Merged ozone profiles from four MIPAS processors

The Michelson Interferometer for Passive Atmospheric Sounding (MIPAS) was an infrared (IR) limb emission spectrometer on the Envisat platform. Currently, there are four MIPAS ozone data products, including the operational Level-2 ozone product processed at ESA, with the scientific prototype processor being operated at IFAC Florence, and three independent research products developed by the Istituto di Fisica Applicata Nello Carrara (ISACCNR)/University of Bologna, Oxford University, and the Karlsruhe Institute of Technology–Institute of Meteorology and Climate Research/Instituto de Astrofísica de Andalucía (KIT–IMK/IAA). Here we present a dataset of ozone vertical profiles obtained by merging ozone retrievals from four independent Level-2 MIPAS processors. We also discuss the advantages and the shortcomings of this merged product. As the four processors retrieve ozone in different parts of the spectra (microwindows), the source measurements can be considered as nearly independent with respect to measurement noise. Hence, the information content of the merged product is greater and the precision is better than those of any parent (source) dataset. The merging is performed on a profile per profile basis. Parent ozone profiles are weighted based on the corresponding error covariance matrices; the error correlations between different profile levels are taken into account. The intercorrelations between the processors’ errors are evaluated statistically and are used in the merging. The height range of the merged product is 20–55 km, and error covariance matrices are provided as diagnostics. Validation of the merged dataset is performed by comparison with ozone profiles from ACEFTS (Atmospheric Chemistry Experiment–Fourier Transform Spectrometer) and MLS (Microwave Limb Sounder). Even though the merging is not supposed to remove the biases of the parent datasets, around the ozone volume mixing ratio peak the merged product is found to have a smaller (up to 0.1 ppmv) bias with respect to ACE-FTS than any of the parent datasets. The bias with respect to MLS is of the order of 0.15 ppmv at 20–30 km height and up to 0.45 ppmv at larger altitudes. The agreement between the merged data MIPAS dataset with ACE-FTS is better than that with MLS. This is, however, the case for all parent processors as well.


Introduction
The Michelson Interferometer for Passive Atmospheric Sounding (MIPAS) was an infrared (IR) limb emission spectrometer onboard the ENVISAT platform.It measured during day and night at 6 to 70 km (up to 170 km in special modes), pole-to-pole, producing more than 1000 profiles per day.Around 30 species, temperature, and cloud composition could be derived from these measurements.In 2002In -2004, the instrument operated in full spectral resolution, with a vertical resolution of about 3.5-6 km for the retrieved ozone Published by Copernicus Publications on behalf of the European Geosciences Union.
product; this period of MIPAS operations is referred to as the full resolution (FR) period.Due to a failure of the instrument's mirror slide in 2004, the operations were suspended during almost a year and were resumed in 2005 with reduced spectral, but improved vertical, resolution.The corresponding period, until the loss of communications with the EN-VISAT platform in April 2012, is referred to as the reduced resolution (RR) period of MIPAS operations.
MIPAS Level-2 data are operationally processed at ESA, with the scientific prototype processor at IFAC Florence (Raspollini et al., 2013).Beyond this, there are three independent scientific Level-2 processors: at the Istituto di Fisica Applicata Nello Carrara (ISAC-CNR)/University of Bologna (Carlotti et al., 2006;Dinelli et al., 2010), at Oxford University (http://www.atm.ox.ac.uk/MORSE/), and the Institute of Meteorology and Climate Research/Instituto de Astrofísica de Andalucía (IMK/IAA) processor at KIT, Karlsruhe Institute of Technology (von Clarmann et al., 2003, 2009).Henceforth, the four processors will be referred to as the ESA, Bologna, Oxford, and KIT processors.The existence of these four products often leads to confusion in the scientific community about their differences and which one to use.The unified description of the four processors is given in Laeng et al. (2015).The main similarities and differences between the four processors can be summarized as follows: all four processors use the same Level-1b spectra provided by ESA, but the Level-2 retrieval algorithms are different; all four processors use microwindows instead of the full spectrum, but microwindow selection differs; for the rationale behind this approach see von Clarmann and Echle (1998), Echle et al. (2000), andDudhia et al. (2002); all four processors apply a global fit approach in a sense that the tangent altitudes of a limb scan are processed simultaneously rather than sequentially (Carlotti, 1988); however, they use different regularization approaches; the Bologna processor uses a full 2-D-approach, that is, all measurements in a complete orbit are processed simultaneously; the horizontal variation of the atmospheric state within the orbit plane is considered; the KIT processor accounts for horizontal temperature gradients along the line of sight direction in the ozone retrieval; the other processors consider atmospheric variation in the altitude domain only.
In the frame of ESA's Ozone Climate Change Initiative project, a round robin evaluation of ozone products from the four MIPAS processors was performed.Comparison with ground-based instruments revealed that all four processors reproduce on average the correct ozone distribution in a similar way, with small differences in bias appearing most clearly in the troposphere.The KIT bias was shown to be less than ±25 ppbv, depending on latitude, compared to a systematic positive bias of roughly +50 ppbv for the other products.In the stratosphere, all four products are positively biased by 2-5 % relative to ozonesonde, lidar, Aura-MLS (Aura-Microwave Limb Sounder), and ACE-FTS (Atmospheric Chemistry Experiment-Fourier Transform Spectrometer) observations.Comparison with satellite instruments showed that the KIT product is biased high only above 35 km, while the other products start to be biased high at somewhat lower altitudes.This was tentatively attributed to the use of the MIPAS AB band (1020-1170 cm −1 ) which is more restrictively used by the KIT processor below 35 km.The MIPAS datasets are generally in better mutual agreement than each of them with other ozone records.The good resemblance of the biases, for example, is a strong indication that the biases are not caused by one particular retrieval processor but come from the instrument itself, from the Level-1 processing or from the spectroscopic data used in the Level-2 retrievals.The details of this analysis are reported in Laeng et al. (2015).Given the availability of four independent MI-PAS ozone data products, the question arose of how to optimize the use of all MIPAS data products.This gave rise to the title activity of this paper: to demonstrate the feasibility of merging the ozone data from the four MIPAS datasets.Contrary to the merging of data from multiple sensors, the following issues do not apply to the merging of multiple data products of a single sensor: sampling issues in space and time, and insufficient time overlap.Further, as follows from Kleinert et al. (2007), the degradation of the MIPAS instrument in the parts of the spectra used by different MI-PAS processors is similar.Since different processors analyze different parts of the spectrum (microwindows), the source (parent) measurements can be considered as nearly independent with respect to the primary measurement errors.The microwindows used by the four MIPAS processors are reported in the Table 1.
The use of different data points by the four processors gives rise to the expectation for the merged product to have a better precision than the individual contributing datasets.However, it should be treated with caution: this expectation relies on the assumption that the dominating source of uncertainties coming with the data is measurement noise, or any other source of random error which is uncorrelated between the parent datasets.It is, of course, not expected that systematic error components will necessarily average out by the merging operation.The merging was performed on the MI-PAS data provided for the round robin exercise only, namely ozone data for the years 2007 and 2008.

Merging approach
The merging is performed on a profile per profile basis.Parent ozone profiles are weighted based on their corresponding random error covariance matrices; the correlations be- tween different altitude levels of the profiles are taken into account.The intercorrelations between the processors' errors are evaluated statistically and are used in the merging.
The small sample size (four processors only) is an obstacle to the identification of outliers.It only takes one processor to significantly deviate from the true profile, and the merged product will be worse than any of the other three.Our choice is, however, to always use all four processors' values.
The merged profile is constructed as a weighted mean of the four parent profiles.For each processor, the errors at different height levels are correlated because of error propagation patterns typical for limb sounding.Therefore, the value of the merged profile at each level is a linear combination of all the levels of all four processors, with weights defined by corresponding error covariance matrices.This means that the weights depend on the uncertainties' size: the smaller the error of a processor are, the larger its contribution to the merged profile is.The merging is performed on a fixed pressure grid which corresponds approximately to the MIPAS RR nominal tangent altitude grid.At the upper and lower ends of the profiles, it occurs frequently that not all four processors provide data.The height range was hence limited to 62-0.8 hPa (∼ 20-55 km).We note n as the number of vertical levels in the profile.The merged profile is obtained as where e is n × n identity matrix, x i , i = 1, 2, 3, 4 is a column vector representing the profile from the processor i, and C is the processor intercorrelation 4n × 4n matrix defined as follows: where S x i is the random retrieval error covariance matrix of processor i, S ij (i and j correspond to processors) are n × n matrices defined by with r ij being n × n matrices representing the correlation of errors on different levels of two processors; the calculation of r ij is discussed below in Sect.3. The random error covariance matrix of the merged profile is given by As the vertical resolutions of the four processors are very close (see Laeng et al., 2015 for details), our choice is not to take the averaging kernels into the merging formalism.See Ceccherini et al. (2015) for a merging formalism involving the averaging kernels but omitting the correlation between the random errors of the parent datasets.

Correlation coefficients
A retrieved atmospheric profile x can be decomposed into the contribution from the true profile and the contribution from bias and random error: where x is the retrieved profile, x is the true profile, and bias and random are respectively the bias and random components www.atmos-meas-tech.net/10/1511/2017/Atmos.Meas.Tech., 10, 1511-1518, 2017 of the error.We pragmatically define "biases" as the average over time at a given altitude and latitude and "random components" as all other error components which survive the subtraction of the bias.Partly correlated errors contribute both to the bias and the random error component.We expect that the random errors in tendency cancel out by the merging, while the bias component of the error will survive the merging.Our approach does not require that the bias components are isolated beforehand, because we statistically evaluate the inter-processor error correlations using debiased differences, as outlined below.
We evaluate the intercorrelation of random errors of different processors by examining the statistics of differences between each pair of processors in the following way.
The random errors of processor i at height p and of processor j at height q, i, j = 0, 1, 2, 3, p, q = 1, ... , n are deduced from the Eq. ( 5) as By definition, the correlation coefficient x X,Y between random variables X and Y is where cov stands for covariance, E stands for mathematical expectation operator, µ X stands for the expectation values of X, µ Y stands for the expectation values of Y , σ X stands for the standard deviation of X, and σ Y stands for the standard deviation of Y .In our case, the random variables would be the random error of the values of processor i at height p and the random error of the values of processor j at height q (i = 0, 1, 2, 3).Since the aim of our merging operation is to improve the precision, the weight of each parent profile shall be associated with the its precision and not with its accuracy.Thus, we use debiased differences to infer the inter-profile correlation coefficients.N is the number of profiles in the whole 2007-2008 sample.On each geolocation k, k = 1, ..., N, we will use the best estimate of the truth x that we have, namely the mean profile of the four processors on this geolocation; we note it by x k , k = 1, ..., N, and its pth level is noted by x p k .The expectation of X (or Y ) is estimated as the mean debiased difference between the retrieved profile and the truth over all geolocations: where the vector xi,l is the profile retrieved by processor i on the lth geolocation and xp i,l is its pth level; x l is the true profile on the lth geolocation and x p l is its pth level.Then the realization of X − µ X on the kth geolocation is (the true profile x k was replaced by its estimate x k ), and similarly the realization of Y − µ Y on the kth geolocation is We use hence the following estimator of the correlation between the random errors p random,i and q random,j .The correlation coefficient r pq ij between the random errors of processor i at height p and the random errors of processor j at height q is In this formula, the third term in each outer sum is the bias of corresponding processor, by taking it out of the first term we obtain a debiased profile, and the second term in the bracket is the mean around which the variation of debiased profiles is calculated.
Note that the obtained matrices are not symmetric, which is to be expected: there is no reason why the random errors of Bologna at height 20 km and random errors of KIT at height 35 km would be correlated exactly as the random errors of Bologna at height 35 km and random errors of KIT at height 20 km.
Figure 1 shows that the errors are correlated for all six pairs; this correlation is not negligible: with a minimal value of −0.6 and maximum of 0.26.This means that the coefficients r pq ij can not be assumed zero and must be taken into the merging formula.Figure 1 also demonstrates that error correlations between the retrieved profiles from different processors are not mostly due to error correlations in the measured spectra: Oxford and ESA use identical measurements (see Table 1), but the highest correlation is observed in Bologna-Oxford and KIT-Oxford cases.However, the similarities/differences in the retrieval algorithms seem to also play a role, and that could explain some high absolute values of the coefficients.A gain in precision of the merged product can be expected if these correlations are adequately taken into account.

Statistical covariance matrices
In the previous section we have discussed how the interprocessor correlations were diagnosed.Now we turn to the  11).Obtained matrices are not symmetric, which is to be expected.The errors are non-negligibly correlated for all six pairs, which means that the coefficients can not be assumed zero and must be taken into the merging formula.
inter-level error covariances of each single processor.These error covariance matrices are needed for each single processor to construct the processor intercorrelation matrix as given by Eq. ( 2).Only two processors, ESA and KIT, provide the covariance matrices for each profile.For the other two processors, only statistical error covariance matrices can be evaluated empirically.The error covariance matrices are taken into the merging for controlling the weight of each processor in the average.Thus, it is more important to evaluate the covariance matrices in a consistent way than to have a particularly good covariance matrix for a subset of profiles.Therefore, we have decided to use statistical covariance matrices for all four processors.In order to reduce the correlation due to natural atmospheric variability, we calculate it on summer profiles in the 20 • S-20 • N latitude band.The mean of all summer tropical stratospheric profiles of a processor i is noted x i, mean .The error covariance matrix of a processor i is evaluated from the sample of debiased summer tropical stratospheric profiles: Note that this formula can also be obtained from the formula for the correlation of errors by taking i = j and replacing the mean of four profiles at each geolocation by the mean of the processor in the summer tropical stratosphere.The obtained error covariance matrices are shown in Fig. 2. The results obtained are consistent with the error bars validation from Laeng et al. (2015).Merging of various data products from the same instrument is not necessarily supposed to remove the bias of the parent datasets.Instead, it is supposed to ameliorate the precision of the product since the parent processors rely on different spectral information (different microwindows).At heights where the precision of the merged product is better  than the precision of any of the parent datasets, the merging is successful.Figure 4 shows simultaneous comparison of the four parent MIPAS datasets and the merged MIPAS dataset with ACE-FTS version 3.5 ozone dataset, for collocation criteria 5 h and 500 km.In terms of precision, hence, the merging is a success at 20-28 and 39-43 km.At 28-38 km, KIT's precision in terms of the standard deviation of the differences is better than the precision of the merged product.At 44-52 km, ESA's precision is better than the precision of the merged product.All these conclusions are conditional to the assumption that ACE represents the truth.Although the merging is not supposed to remove the bias, it can change the bias through the averaging, and depending on the signs of biases of parent profiles, this could lead to the improvement of the bias.This is what is happening in the comparison with ACE-FTS: at 24-28 and 33-37 km, the merged product agrees with ACE ∼ 1 % better than KIT, while at all other heights, KIT agrees better.Interestingly, in integrated view over the altitude range around the ozone VMR (volume mixing ratio) peak, where all four processors have a known positive bias (Laeng et al., 2015), the merged product is performing ∼ 0.8 % better than any of the four processors (Fig. 4b).
Figure 5 shows simultaneous comparison of the four parent MIPAS datasets and the merged MIPAS dataset with MLS version 3.3 dataset, for collocation criteria 4 h and 250 km.At 20-25 km, the precision of the merged product is better than the precision of any individual dataset.When looking at the whole height range, the overall precision of the merged product is better than the precision of any of the parent datasets.The overall agreement of the merged product with MLS is not as good as that with ACE-FTS; this is also the case for all parent datasets.In terms of bias, the merged product performs ∼ 1.5-2 % better at 24-33 and 41-45 km, while KIT performs better at the remaining heights.In particular, unlike for the comparison with ACE, around the ozone VMR peak, the agreement of KIT is ∼ 2.5-2.8 % better than the agreement of the merged dataset.

Conclusions
We created a 2-year dataset of merged ozone profiles from four independent MIPAS Level-2 processors.The novelty of the product is a mathematically clean way of performing the merging: the weighting of parent profiles is realized by corresponding inverse error covariance matrices, the correlations between different profile levels are considered, and the intercorrelations between processors' errors are evaluated statistically and are used in the merging.In comparison to the individual parent datasets, the merged product has a restricted height range (20-55 km) and only a statistical covariance matrix can be provided.Validation of the merged dataset is performed by comparing with ozone profiles from ACE-FTS and MLS.Comparison with ACE-FTS looks better than with MLS.This is, however, the case for all parent processors as well.Despite the fact that the merging is not supposed to remove the bias, the high bias around the ozone VMR peak known for the parent profiles is reduced in comparison with ACE-FTS (but not with MLS).The overall precision of the merged product is better than that of any of the four processors.This product could therefore be of use in specific studies requiring improved precision of the MIPAS ozone record.

Figure 1 .
Figure 1.Correlation of errors of four processors calculated by Eq. (11).Obtained matrices are not symmetric, which is to be expected.The errors are non-negligibly correlated for all six pairs, which means that the coefficients can not be assumed zero and must be taken into the merging formula.

Figure 2 .
Figure 2. Statistical covariance matrices of four parent MIPAS processors.The white areas in Bologna and Oxford plots are for values bigger than 0.3: up to 0.82 for Bologna and up to 0.44 for Oxford processors.

Figure 3 .
Figure3.Parent MIPAS profiles and the resulting merged MI-PAS profile on geolocation 33441_20080723T072843Z (0.2 • S, 40.5 • E).The profile from the Bologna processor is the blue line; from the ESA processor, red; from the KIT processor, green; and from the Oxford processor, pink.The merged profile is the brown line.

Figure 4 .
Figure 4. Mean profiles (a), bias (b), and precision validation (debiased standard deviation of the differences, c) of the four parent datasets and merged MIPAS dataset with respect to ACE-FTS ozone profiles in 2007-2008.

Figure 5 .
Figure 5. Mean profiles (a), bias (b), and precision validation (debiased standard deviation of the differences, c) of the four parent datasets and merged MIPAS dataset with respect to MLS ozone profiles in 2007-2008.

Table 1 .
Spectral analysis windows used by the four MIPAS processors for ozone retrieval.