4.1.1 Vertical quantity matching

When changing between concentration-type quantities like number density and volume mixing ratio, a diagonal level-by-level unit conversion matrix M can be constructed straightforwardly (e.g., Keppens et al., 2015, Table  B1). The quantity-matching operation for a vertical profile x, with corresponding ex ante covariance matrix S and possibly averaging kernel matrix A, is then easily achieved by matrix multiplication (Keppens et al., 2015):

Note that these operations have no effect on the fractional covariance matrix, nor on the fractional (or logarithmic) averaging kernel matrix that is required for information content studies (see Keppens et al., 2015 and Sect. 5).

When going from a concentration-type representation on levels to one between levels (i.e., on layers, like partial columns), one can choose the integration boundaries either on the given levels or in between them with the exception of the outer edges, resulting in a rectangular or square conversion matrix M, respectively (Keppens et al., 2015, Table B1):

and

with

Here h has been used as a generalization of the vertical coordinate (altitude, pressure or other) with L elements, while u is the relevant unit conversion constant. Note that in contrast with Eq. (9), the inverse of M in Eq. (10) is not under-constrained, which favors the latter at the small price of the need for an h^′.

4.1.2 Vertical sampling matching

The number of levels (for point-like concentration values) or layers (for vertically integrated column values) and their vertical locations or boundaries have to be identical for two profiles to be quantitatively compared. One can opt for an explicit vertical range matching of the two profiles first, e.g., by vertical clipping of the one or by extension by use of a climatology of the other. The latter can be applied when later profile operations require knowledge of the atmospheric state over its full vertical range (Keppens et al., 2015). When vertical range matching is skipped, vertical sampling or grid-matching operations – often called regridding – automatically limit the height range of the input profile grid to its vertical overlap region with the target profile grid.

Several regridding approaches are in use, although their application typically can depend on units and/or the vertical resolution discrepancy between the input and target grids.

• Straightforward regridding by (linear or other) interpolation only works appropriately, i.e., with minimum information loss, when going from a coarser-resolution input grid to a finer-resolution target grid. Although the corresponding interpolation matrix W is not square, it is applied in an identical way as the unit conversion matrix M in Eq. (8):

  $$\begin{array}{}\text{(12a)}& {\displaystyle }{\mathit{x}}^{\prime }& {\displaystyle }=\mathbf{W}\mathit{x},\text{(12b)}& {\displaystyle }{\mathbf{S}}^{\prime }& {\displaystyle }={\mathbf{WSW}}^T,\text{(12c)}& {\displaystyle }{\mathbf{A}}^{\prime }& {\displaystyle }={\mathbf{WAW}}^{*}.\end{array}$$

  As the inverse of a non-square matrix is ill-posed, its definition depends on the norm one wishes to minimize. Opting for a simple least squares difference between the input and target profiles yields ${\mathbf{W}}^{*}=({\mathbf{W}}^T\mathbf{W}{)}^{-\mathrm{1}}{\mathbf{W}}^T$ (Rodgers, 2000). The elements of W are determined by the interpolation function one applies, which introduces an additional term in the uncertainty budget (see Sect. 6).

• In order not to suggest a vertical resolution that is misleadingly much higher than the effective vertical resolution of (one of) the observations, atmospheric state profile comparisons are often made on the vertical grid of the product with the coarsest sampling. When consequently the input grid has a finer resolution than the target grid, one can easily invert the problem by constructing an interpolation matrix W for going from the target grid to the input grid, and then applying the regular regridding formulas for ${\mathbf{W}}^{\prime }={\mathbf{W}}^{*}$. This approach is denoted as pseudo-inverse (linear or other) regridding. When going from a fine to a coarse vertical sampling, this method approximately conserves the atmospheric constituent’s mass (vertically integrated amount) over a wavelet-like vertical window function (also see Sect. 5).

• In practice vertical sampling definitions might change in time, or one might not know beforehand whether the target grid is coarser than the input grid or vice versa, or both grids may be similar. Calisesi et al. (2005) have therefore proposed to combine both of the previous methods by first constructing two interpolation matrices, W₁ and W₂ respectively, going from the input and target grids to a conjoint super grid that is the resorted union of the input and target grids. As a result, one can generally apply vertical sampling matrix ${\mathbf{W}}^{\prime }={\mathbf{W}}_{\mathrm{2}}^{*}{\mathbf{W}}_{\mathrm{1}}$ as before.

• One might instead prefer the total vertical column amount to be conserved during the regridding operation. Such mass-conserved regridding is easily achieved for partial column quantities, whether going from finer to coarser resolution or vice versa. It is sufficient to construct an overlap matrix that contains the fractions of how much each target grid layer is covered by an input grid layer (Langerock et al., 2015). Assuming that the ith output grid layer overlaps with the jth input grid layer, the corresponding element of the conversion matrix is the following interpolation factor

  $$\begin{array}{}\text{(13)}& \begin{array}{rl}\mathbf{W}(i,j)=\mathrm{\Delta }{h}_{\mathrm{in},j}^{-\mathrm{1}}& \left[\min \right(h_{\mathrm{out},i}^U,h_{\mathrm{in},j}^U)\\ -& \max (h_{\mathrm{out},i}^L,h_{\mathrm{in},j}^L)],\end{array}\end{array}$$

  with Δh_in,j the input layer thickness and the indices U and L indicating the layers' upper and lower height bounds, respectively. Note that this expression implicitly makes use of the conjoint super grid (see previous), allowing broad usage of this approach. If there is no overlap between target layer i and input layer j, then W(i,j) equals 0. The coefficients of the conversion matrix therefore satisfy $\mathrm{0}\le \mathbf{W}(i,j)\le \mathrm{1}$.

• Total mass-conserved regridding of concentration-type quantities defined on vertical levels or as vertical averages, as is often the case in model fields, is somewhat less straightforward. Before being able to apply the conversion matrix as defined in the previous expression, the point-like concentration values of the input profile must be converted to vertically integrated values, and after the subsequent mass-conserved regridding operation a conversion to the initial units is needed. A combination of Eq. (13) with a forward and backward conversion by use of Eq. (8) including M defined by Eq. (10) (with well-defined inverse) is hence required, although this can be achieved in arbitrary units (i.e., without the need for the unit conversion constant u):

  $$\begin{array}{}\text{(14)}& {\mathbf{W}}^{\prime }={\mathbf{M}}_{\mathrm{out}}^{*}{\mathbf{WM}}_{\mathrm{in}}.\end{array}$$

  Here M_in and M_out are the conversion matrices for the input and output grids to their layer representations, respectively, and W is the regular mass-conserved regridding matrix of Eq. (13).

4.3.1 Imposing retrieval artifacts

• Measurement weight matching. The vertical sensitivity of an atmospheric state retrieval is defined as the column vector of its averaging kernel row sums. It is given by Au if u represents the vertical unit vector and can be considered an estimator of the height-dependent fraction of the retrieval that comes from the measurement, rather than from the prior profile (Rodgers, 2000). One can thus define a diagonal measurement weight matrix W^M (or prior weight matrix I−W^M) by diag(W^M)=Au, so that A=W^MA¹ or ${\mathbf{A}}^{\mathrm{1}}=({\mathbf{W}}^{\mathrm{M}}{)}^{-\mathrm{1}}\mathbf{A}$. The last expression provides the most straightforward calculation of the AK-based vertical smoothing matrix V in the previous section. The measurement weight harmonization operation that matches the sensitivity of an atmospheric state with the measurement weight W^M of a given retrieved state is thus given by ${\mathit{x}}^{\prime }={\mathbf{W}}^{\mathrm{M}}\mathit{x}$, with ${\mathbf{A}}^{\prime }={\mathbf{W}}^{\mathrm{M}}\mathbf{A}$. Here ${\mathbf{S}}^{\prime }=\mathbf{S}$, as the diagonal matrix W^M can be considered as merely a vertically resolved conversion constant.

• Prior matching. Rodgers (2000, Eq. 10.48) provides an expression for replacing the prior constraint R_a and profile shape x_a within a given retrieval by ${\mathbf{R}}_a^{\prime }$ and ${\mathit{x}}_a^{\prime }$, respectively:

  $$\begin{array}{}\text{(15)}& {\mathit{x}}^{\prime }=({\mathbf{S}}^{-\mathrm{1}}-{\mathbf{R}}_a+{\mathbf{R}}_a^{\prime }{)}^{-\mathrm{1}}({\mathbf{S}}^{-\mathrm{1}}\mathit{x}-{\mathbf{R}}_a{\mathit{x}}_a+{\mathbf{R}}_a^{\prime }{\mathit{x}}_a^{\prime }),\end{array}$$

  whereby the prior constraint is typically, but not necessarily, given by the inverse of the prior covariance matrix, ${\mathbf{R}}_a={\mathbf{S}}_a^{-\mathrm{1}}$. If only the prior’s profile shape x_a is substituted by ${\mathit{x}}_a^{\prime }$ (i.e., for ${\mathbf{R}}_a^{\prime }={\mathbf{R}}_a$), the prior matching formula simplifies to the rather intuitive (Rodgers and Connor, 2003, Eq. 10)

  $$\begin{array}{}\text{(16)}& {\mathit{x}}^{\prime }=\mathit{x}-(\mathbf{I}-\mathbf{A})({\mathit{x}}_a-{\mathit{x}}_a^{\prime }),\end{array}$$

  by taking into account that $\mathbf{I}-\mathbf{A}={\mathbf{SS}}_a^{-\mathrm{1}}$ (Rodgers, 2000, Eq. 2.79). The latter moreover shows that ${\mathbf{A}}^{\prime }=\mathbf{A}+{\mathbf{SS}}_a^{-\mathrm{1}}-{\mathbf{S}}^{\prime }{{\mathbf{S}}_a^{\prime }}^{-\mathrm{1}}$. The prior-changed covariance matrix S^′ is obtained by substituting S_a by ${\mathbf{S}}_a^{\prime }$ in the retrieval's expression for S (for example Rodgers, 2000, Eq. 2.27): ${\mathbf{S}}^{\prime }=({\mathbf{S}}^{-\mathrm{1}}-{\mathbf{S}}_a^{-\mathrm{1}}+{{\mathbf{S}}_a^{\prime }}^{-\mathrm{1}}{)}^{-\mathrm{1}}$.

• Re-optimized prior matching. By changing the prior in a given retrieval, the resulting atmospheric state profile x^′ and its AKM will no longer provide an optimally estimated (i.e., with minimal retrieval gain function) representation with respect to the new constraint. Hence re-optimization of the prior-matched profile might be required. When ${\mathbf{S}}_a^{\prime }={\mathbf{S}}_a$, this can be achieved by (Rodgers and Connor, 2003, Eq. 18)

  $$\begin{array}{}\text{(17)}& {\mathit{x}}^{\prime }={\mathit{x}}_a^{\prime }+{\mathbf{S}}_a^{\prime }{\mathbf{A}}^T({\mathbf{AS}}_a^{\prime }{\mathbf{A}}^T+\mathbf{S}{)}^{-\mathrm{1}}(\mathit{x}-{\mathit{x}}_a^{\prime }),\end{array}$$

  whereby the atmospheric state vector x on the right-hand side is taken from the output of the prior matching operation in Eq. (16). The re-optimized prior matching that combines Eqs. (16) and (17) thus takes the form ${\mathit{x}}^{\prime }=\mathbf{P}[\mathit{x}-(\mathbf{I}-\mathbf{A}\left){\mathit{x}}_a\right]+(\mathbf{I}-{\mathbf{A}}^{\prime }){\mathit{x}}_a^{\prime }$, with $\mathbf{P}={\mathbf{S}}_a^{\prime }{\mathbf{A}}^T({\mathbf{AS}}_a^{\prime }{\mathbf{A}}^T+\mathbf{S}{)}^{-\mathrm{1}}$ and ${\mathbf{A}}^{\prime }=\mathbf{PA}$ like a vertical smoothing operation. Just as before, ${\mathbf{S}}^{\prime }={\mathbf{PSP}}^T$ correspondingly. Ridolfi et al. (2006, Eq. 8) obtained the same conversion matrix P by constructing an optimal interpolation method, i.e., an optimization through trace(S_Δ) minimization of combined vertical sampling and smoothing matching operations (hence the non-square AKMs and preceding A₁ in their expression). However, based on the complete data fusion framework, Ceccherini et al. have been able to construct a more general re-optimized prior matching operation that is valid for all new prior profile shapes and constraints (Ceccherini et al., 2014, Eq. 7):

  $$\begin{array}{}\text{(18)}& {\mathit{x}}^{\prime }=\mathbf{P}[\mathit{x}-(\mathbf{I}-\mathbf{A}\left){\mathit{x}}_a\right]+\mathbf{PS}({\mathbf{A}}^T{)}^{-\mathrm{1}}{\mathbf{R}}_a^{\prime }{\mathit{x}}_a^{\prime }.\end{array}$$

  This expression even holds when ${\mathbf{R}}_a^{\prime }$ and ${\mathit{x}}_a^{\prime }$ are defined on a different vertical grid than the input profile x. In that case it is sufficient to replace A by AW^* in Eq. (18) (also in P) with W, a regridding matrix as defined in Sect. 4.1.2 (Ceccherini et al., 2018).

• Averaging kernel smoothing. In practice the covariance matrices that are needed in Eqs. (17) and (18) are not always provided to data users, or implementation of the (re-optimized) prior matching is not preferred. One can however avoid these operations by equalling ${\mathit{x}}_a^{\prime }$ to the prior of one of the profiles in a comparison and then applying a vertical smoothing matching and measurement weight matching on this profile by use of the second profile’s averaging kernel matrix. In doing so, only the second profile has to be prior-corrected, resulting in only one non-optimal representation, while this non-optimality and the initial prior constraint of the second profile are enforced on the first profile and therefore drop out of the difference comparison. This whole process thus combines vertical smoothing matching with V=A¹, measurement weight matching with diag(W^M)=Au, and prior matching that does not require re-optimization (Eq. 16). By, for example, comparing a vertically smoothed and measurement weight-corrected reference profile ${\mathit{x}}_{\mathrm{r}}^{\prime }={\mathbf{W}}_{\mathrm{s}}^M{\mathbf{A}}_{\mathrm{s}}^{\mathrm{1}}{\mathit{x}}_{\mathrm{r}}={\mathbf{A}}_{\mathrm{s}}{\mathit{x}}_{\mathrm{r}}$ with a prior shape-corrected profile under study ${\mathit{x}}_{\mathrm{s}}^{\prime }={\mathit{x}}_{\mathrm{s}}-(\mathbf{I}-{\mathbf{A}}_{\mathrm{s}})({\mathit{x}}_{\mathrm{a},\mathrm{s}}-{\mathit{x}}_{\mathrm{a},\mathrm{r}})$ one obtains (omitting any necessary vertical representation matching)

  $$\begin{array}{}\text{(19)}& \begin{array}{rl}\mathbf{\Delta }x& ={\mathit{x}}_{\mathrm{s}}^{\prime }-{\mathit{x}}_{\mathrm{r}}^{\prime }\\ & ={\mathit{x}}_{\mathrm{s}}-(\mathbf{I}-{\mathbf{A}}_{\mathrm{s}})({\mathit{x}}_{\mathrm{a},\mathrm{s}}-{\mathit{x}}_{\mathrm{a},\mathrm{r}})-{\mathbf{W}}_{\mathrm{s}}^M{\mathbf{A}}_{\mathrm{s}}^{\mathrm{1}}{\mathit{x}}_{\mathrm{r}}\\ & ={\mathit{x}}_{\mathrm{s}}-[{\mathbf{A}}_{\mathrm{s}}{\mathit{x}}_{\mathrm{r}}+(\mathbf{I}-{\mathbf{A}}_{\mathrm{s}}\left)\right({\mathit{x}}_{\mathrm{a},\mathrm{s}}-{\mathit{x}}_{\mathrm{a},\mathrm{r}}\left)\right].\end{array}\end{array}$$

  If additionally the reference profile results from an in situ measurement or model (${\mathit{x}}_{\mathrm{a},\mathrm{r}}=\mathbf{0}$), this equation can just as well be inferred by considering ${\mathit{x}}_{\mathrm{s}}^{\prime }={\mathit{x}}_{\mathrm{s}}$ and imposing the satellite retrieval on the reference profile, meaning that the unknown true profile x_t is replaced by x_r in Eq. (4) (without the error term):

  $$\begin{array}{}\text{(20)}& {\mathit{x}}_{\mathrm{r}}^{\prime }={\mathbf{A}}_{\mathrm{s}}{\mathit{x}}_{\mathrm{r}}+(\mathbf{I}-{\mathbf{A}}_{\mathrm{s}}){\mathit{x}}_{\mathrm{a},\mathrm{s}}.\end{array}$$

  It is typically the latter interpretation that is referred to as averaging kernel smoothing (of x_r). The term however also applies when this reference profile is a retrieved product as well. In that case one can even apply symmetrical smoothing of both the satellite and reference profiles if they show comparable vertical smoothing (Rodgers and Connor, 2003; von Clarmann and Grabowski, 2007)

  $$\begin{array}{}\text{(21)}& \begin{array}{rl}\mathbf{\Delta }x=& {\mathbf{A}}_{\mathrm{r}}[{\mathit{x}}_{\mathrm{s}}-(\mathbf{I}-{\mathbf{A}}_{\mathrm{s}}\left)\right({\mathit{x}}_{\mathrm{a},\mathrm{s}}-{\mathit{x}}_{\mathrm{a},\mathrm{c}}\left)\right]\\ & -{\mathbf{A}}_{\mathrm{s}}[{\mathit{x}}_{\mathrm{r}}-(\mathbf{I}-{\mathbf{A}}_{\mathrm{r}}\left)\right({\mathit{x}}_{\mathrm{a},\mathrm{r}}-{\mathit{x}}_{\mathrm{a},\mathrm{c}}\left)\right]\end{array}\end{array}$$

  for prior matching to a common x_a,c. This expression simplifies to Eq. (19) for ${\mathit{x}}_{\mathrm{a},\mathrm{c}}={\mathit{x}}_{\mathrm{a},\mathrm{r}}$ and A_r=I.

Figure 1Profile harmonization flowchart, indicating the order of the matching operations outlined in the text. Rectangular boxes are optional, while hexagons are mandatory. The maximum likelihood (ML) representation has here been included as a prior matching operation with ${\mathbf{R}}_a^{\prime }=\mathbf{0}$.

Download

4.3.2 Removing retrieval artifacts

• Maximum likelihood representation. The maximum likelihood representation (MLR) of a retrieved atmospheric state profile corresponds to the retrieval in the absence of explicit prior information, i.e., the retrieval for R_a=0 (Rodgers, 2000). One can thus easily convert a given retrieved profile to its maximum likelihood representation by performing a prior matching operation as in Eq. (15) with ${\mathbf{R}}_a^{\prime }=\mathbf{0}$ (Rodgers, 2000; von Clarmann et al., 2015):

  $$\begin{array}{}\text{(22)}& {\mathit{x}}^{\prime }=({\mathbf{S}}^{-\mathrm{1}}-{\mathbf{R}}_a{)}^{-\mathrm{1}}({\mathbf{S}}^{-\mathrm{1}}\mathit{x}-{\mathbf{R}}_a{\mathit{x}}_a).\end{array}$$

  The resulting covariance matrix is given by ${\mathbf{S}}^{\prime }=({\mathbf{S}}^{-\mathrm{1}}-{\mathbf{S}}_a^{-\mathrm{1}}{)}^{-\mathrm{1}}$, while the averaging kernel matrix becomes the unit matrix, making re-optimization meaningless. This does however not mean that the MLR is fully unconstrained, as it is still implicitly constrained by its vertical grid and the related interpolation convention (von Clarmann and Grabowski, 2007).

• Information-centered representation. In order to explicitly remove all prior information from a given retrieval and hence simulate a direct measurement with all levels or layers representing one degree of freedom, the prior constraint replacement operation has to be combined with a vertical regridding operation while also setting ${\mathit{x}}_a^{\prime }=\mathbf{0}$ (von Clarmann and Grabowski, 2007):

  $$\begin{array}{}\text{(23)}& {\mathit{x}}^{\prime }=\mathbf{W}({\mathbf{S}}^{-\mathrm{1}}-{\mathbf{R}}_a+{\mathbf{R}}_a^{\prime }{)}^{-\mathrm{1}}({\mathbf{S}}^{-\mathrm{1}}\mathit{x}-{\mathbf{R}}_a{\mathit{x}}_a).\end{array}$$

  By insertion of the transposed regridding matrix and its pseudo-inverse, one obtains

  $$\begin{array}{}\text{(24)}& \begin{array}{rl}{\mathit{x}}^{\prime }& =\mathbf{W}({\mathbf{S}}^{-\mathrm{1}}-{\mathbf{R}}_a\\ & +{\mathbf{R}}_a^{\prime }{)}^{-\mathrm{1}}{\mathbf{W}}^T{\mathbf{W}}^{*T}({\mathbf{S}}^{-\mathrm{1}}\mathit{x}-{\mathbf{R}}_a{\mathit{x}}_a)\\ & =({\mathbf{W}}^{*T}{\mathbf{S}}^{-\mathrm{1}}{\mathbf{W}}^{*}-{\mathbf{W}}^{*T}{\mathbf{R}}_a{\mathbf{W}}^{*}\\ & +{\mathbf{W}}^{*T}{\mathbf{R}}_a^{\prime }{\mathbf{W}}^{*}{)}^{-\mathrm{1}}{\mathbf{W}}^{*T}({\mathbf{S}}^{-\mathrm{1}}\mathit{x}-{\mathbf{R}}_a{\mathit{x}}_a).\end{array}\end{array}$$

  In order to remove all prior information from the retrieval outcome, one thus has to determine W and ${\mathbf{R}}_a^{\prime }$ that impose the hard constraint ${\mathbf{W}}^{*T}{\mathbf{R}}_a^{\prime }{\mathbf{W}}^{*}=\mathbf{0}$ non-trivially instead of using the soft MLR constraint ${\mathbf{R}}_a^{\prime }=\mathbf{0}$ (von Clarmann and Grabowski, 2007). The difficulty of this approach lies in the determination of these two matrices in agreement with (i.e., causing minimal loss) the number of independent pieces of information or degrees of freedom in the initial measurement, which is given by trace(A). The study from von Clarmann and Grabowski (2007) provides methods to do so – in both staircase and triangular representation – that are rather extensive and therefore not reproduced here. Equation (24) can also be obtained from Eq. (18) including W^*, while it is in agreement with Rodgers (2000, Eq. 10.50) only if the latter’s back-transformation to the original grid is omitted. Rodgers therefore still indicates his representation as a maximum-likelihood solution, while here the term information-centered representation by von Clarmann and Grabowski (2007) is adopted. Note however that these two references have considered opposite directions in the definition of their respective regridding matrices W. Again ${\mathbf{A}}^{\prime }=\mathbf{I}$, while the covariance matrix is now given by ${\mathbf{S}}^{\prime }=\mathbf{W}({\mathbf{S}}^{-\mathrm{1}}-{\mathbf{S}}_a^{-\mathrm{1}}{)}^{-\mathrm{1}}{\mathbf{W}}^T$ in agreement with the prior matching expression upon addition of a regridding operation.