2.1.1 AVHRR

AVHRR is a cross-track scanner with a 2900 km swath width, providing almost daily global coverage. The sensor is equipped with six spectral channels (Table 1), out of which only five can be transmitted simultaneously so that either channel 3a or 3b is available. In-flight calibration is performed only for thermal channels, using a stable blackbody and a space view as references. AVHRR has been mounted on several NOAA platforms as well as on EUMETSAT's Metop-A/B, all of which are sun-synchronous, polar-orbiting satellites. Due to a lack of orbit control technology for all NOAA AVHRRs, there is considerable orbit drift in equatorial crossing times (ECT) for both morning (ECT < 12:00 local solar time, LST) and afternoon (ECT > 12:00 LST) satellites. To reduce drift-induced changes in retrieved cloud properties, any AVHRR is replaced with its corresponding successor once available (the AVHRR prime record). Typically, one morning and one afternoon NOAA satellite are in orbit at any time.

Table 1The CC4CL AVHRR heritage dataset channel characteristics for AVHRR, AATSR, and MODIS. Instrument noise as applied within CC4CL is in reflectance for CC4CL channels 1–3 and in brightness temperature (K) for channels 4–6.

Download Print Version | Download XLSX

For CC4CL, we use Global Area Coverage (GAC) L1c data on a reduced spatial resolution of 1.1 km × 4 km at nadir (Devasthale et al., 2018). The AVHRR GAC L1c data record, including advanced intercalibration efforts, was produced for ESA Cloud_cci and CMSAF (Karlsson et al., 2013; Schulz et al., 2009). CC4CL processed AVHRR data from August 1981 (NOAA-7) to December 2014 (Metop-A + NOAA-19). We applied a filtering technique to channel 3b data and a database algorithm for splitting midnight orbits and blacklisting.

2.1.2 MODIS

MODIS is carried by NASA's Terra and Aqua satellite platforms in a near sun-synchronous polar orbit at 705 km altitude. Due to orbit control, ECT is a constant 10:30 LST for Terra and 13:30 LST for Aqua. The Aqua satellite is a member of the A-Train constellation, which also includes the CALIPSO and CloudSat satellites. MODIS is a cross-track scanner with a 2330 km swath width, producing a complete near-global coverage in less than 2 days (Xiong et al., 2009).

CC4CL is applied to C6 MOD021km (Terra) and MYD021km (Aqua) L1b input data (NASA LP DAAC, 2015). For the AVHRR heritage dataset produced here, the NASA Goddard Space Flight Center performed a spectral subsetting of the 36 MODIS channels available (see Table 1 for the channels extracted), and data were directly shipped to ECMWF (European Centre for Medium-Range Weather Forecasts) for archiving. The files are stored in HDF-EOS format at 1 km spatial resolution, with the 250 and 500 m channels having been aggregated to 1 km resolution. MODIS L1b data are organized in granules, each of which contains ∼ 5 min of MODIS data or ∼ 203 scan lines. Geolocation information is provided in separate files for Terra (MOD03) and Aqua (MYD03), containing geodetic latitude and longitude and solar–satellite zenith and azimuth angles. L1b data are corrected for all known instrumental effects through on-board calibrator data and are organized into a viewing swath matching the geolocation file structure (MODIS Characterization Support Team, 2009). With CC4CL, we processed data from February 2000 (Terra) or August 2002 (Aqua) to December 2014.

2.1.3 ATSR-2 and AATSR

The second-generation ATSR-2 and third-generation AATSR (Merchant et al., 2012) were launched on ESA's polar-orbiting satellites ERS-2 and ENVISAT in April 1995 and March 2002, respectively. Both platforms were put into a sun-synchronous orbit at ∼ 780 km altitude, with ECT of 10:30 LST for ERS-2 and ECT of 10:00 for ENVISAT. Both ATSRs are identical in their overall configuration except for data transfer bandwidth (Table 1). ATSR is equipped with on-board calibration capabilities, such as two blackbody targets for the thermal channels and a sun-illuminated opal target for the visible–near-infrared channels. ATSR uses a dual-view system: a nadir view and a forward view that scans the surface at an angle of 55^∘. The continuous scanning pattern produces a nadir resolution of approximately 1 km × 1 km with a swath width of 512 pixels or ∼ 500 km, providing global coverage every 6 days.

We used no forward view data for cloud retrievals, as the three-dimensional cloud structure produces parallax effects which are not accounted for within the current forward model. With CC4CL, we processed ATSR data from launch date until May 2003 (ERS-2) and April 2012 (ENVISAT).

2.2.1 ERA-Interim

We use ERA-Interim data as first-guess input for the retrieval of surface temperature and as input to a neural network cloud mask (see Sect. 3.3.1). ERA-Interim is a reanalysis of the global atmosphere and is available from 1979 until today (Berrisford et al., 2011; Dee et al., 2011). The atmospheric profile variables are defined at 60 vertical levels. The original horizontal resolution is defined through a T255 spherical-harmonic representation for the basic dynamical fields and through a reduced Gaussian grid with ∼ 79 km spacing for surface fields. We downloaded ERA-Interim data from the ECMWF's MARS archive at a spatial resolution of 0.72^∘ (the default preprocessing grid resolution) and at a higher resolution of 0.1^∘ for the neural network cloud mask input variables (Table A1). We acquired analysis (i.e. not forecast) data at 6-hourly timesteps. After download, all files were remapped to the CC4CL preprocessor grid through Climate Data Operators (CDO, 2015). This was necessary, as ERA-Interim coordinates are defined at the cell boundaries, whereas they are defined at the cell centres within CC4CL. The reanalysis data are temporally interpolated onto the satellite image's centre time by linearly weighting the files before and after.

ERA-Interim's land surface model still needs to be improved in terms of its simulation of soil hydrology and snow cover. This affects the utilization of satellite data over land surfaces within ERA-Interim, which has negative effects on the representation of clouds and precipitation (Berrisford et al., 2011). The confidence in temperature trend estimates, however, has improved considerably so that ERA-Interim data have been used as an alternative to observational datasets to monitor climate change (Willett et al., 2010).

2.2.2 Land use

We downloaded United States Geological Service (USGS) Land Use/Land Cover raster data from the global land cover characteristics database (US Geological Survey, 2016). This was necessary, as early AVHRR data are distributed without masking information. The USGS data are used as a land–sea mask within the optimal estimation retrieval (Sect. 3.3.3) as well as a land cover classification within the cloud mask and the Pavolonis cloud typing scheme (Sect. 3.3.2). The dataset is defined on a regular latitude–longitude grid with 0.05^∘ resolution. The USGS land cover classification was primarily derived from 1 km AVHRR Normalized Difference Vegetation Index (NDVI) 10-day composites for April 1992 through March 1993 (US Geological Survey, 2016).

2.2.3 Land surface BRDF

MODIS C6 bidirectional reflectance distribution function (BRDF) data (MCD43C1; Schaaf and Wang, 2015), providing kernel weights for the Ross thick–Li sparse reciprocal BRDF model, are used within the retrieval scheme to set surface albedo and bidirectional reflectance distribution conditions. These data are available every 8 days derived from cloud-cleared 16-day Terra and Aqua measurements, and provided in HDF-EOS format at 0.05^∘ spatial resolution. MCD43C1 data are classified as high-quality given sufficient observations, and otherwise a low-quality estimate is produced based on climatology anisotropy models. Validation against albedo measurements made at Baseline Surface Radiation Network (BSRN) sites show that the black-sky and white-sky albedo computed from the single sensor MCD43A1 high-quality product are well within 5 % of the measured albedo, while the low-quality product is within 10 % (Lucht, 1998).

We regridded MCD43C1 data to instrument resolution through bilinear interpolation, and filled missing pixels within the time series with pixel values of the temporally closest 8-day composite file providing valid data. For the pre-MODIS era, we produced a BRDF climatology by averaging all data available for a particular 8-day time slot. MCD43C1 kernel weights are applied to all CC4CL sensors, neglecting differences in spectral response functions. This might result in retrieval biases, particularly in spectral regions that are sensitive to rapidly changing environmental processes such as vegetation growth (near-infrared). Note that the use of a climatology would add a discontinuity in the surface time series if there were trends in the surface BRDF and emissivity time series during the MODIS era.

2.2.4 Land surface emissivity

For land surface emissivity, we used the Cooperative Institute for Meteorological Satellite Studies (CIMSS) global land surface infrared emissivity database created by the baseline fit method (Seemann et al., 2008). These data are derived from the MODIS operational land surface emissivity product (MOD11), to which the fit method is applied for filling spectral gaps between channels. CIMSS emissivity data are available on a monthly basis at 10 wavelengths with 0.05^∘ spatial resolution.

As for BRDF, we produced a land surface emissivity climatology for the pre-MODIS era by averaging all data available for a particular month.

3.3.1 Cloud detection

The CC4CL cloud mask is produced by (1) estimating pseudo-CALIOP cloud optical depth (ANNCOD) from L1 measurements with an artificial neural network (ANN), (2) correcting ANNCOD for viewing-angle dependencies, and (3) classifying ANNCOD into binary cloud mask information by thresholding.

CC4CL applies a set of ANNs for cloud masking, one for each of the illumination conditions day (solar zenith angle ${\mathit{\theta }}_{\mathrm{0}}<\mathrm{80}{}^{\circ }$), night (${\mathit{\theta }}_{\mathrm{0}}\ge \mathrm{90}{}^{\circ }$), and twilight ($\mathrm{80}\le {\mathit{\theta }}_{\mathrm{0}}<\mathrm{90}{}^{\circ }$). The ANNs are multilayer perceptrons with one input layer, one hidden layer with 50 neurons, and one output layer, which produces ANNCOD ranging from 0 to 1. Through incremental testing, we found that 50 neurons was the value for which the trade-off between output quality and computing speed was optimal. For the input layer, input variables are surface temperature, snow or ice cover, and the land–sea mask for all three cloud masks. Regarding sensor data, input channels are Ch1, Ch2, Ch5, Ch6, and Ch5-Ch6 for the day ANN, Ch4, Ch5, Ch6, Ch5-Ch4, and Ch5-Ch6 for the night ANN, and Ch5, Ch6, and Ch5-Ch6 for the twilight ANN.

The various ANNs were trained with NOAA-18 AVHRR L1c data, ancillary information (ECMWF land–sea mask, snow–ice mask, and surface temperature), and cloud optical depth (COD) “truth” data obtained from CALIOP'S 532 nm lidar product (CAL_LID_L2_05kmCLay-Prov-V3-01). AVHRR Ch3a data were generally excluded. We trained the day ANN with all remaining AVHRR channels, but also excluded Ch3b to be consistent with those NOAA platforms that switch between Ch3b transmission at night and Ch3a during the day (NOAA-16, NOAA-17, Metop-A). For night and twilight conditions, we produced ANNs both with and without Ch3b data input. This was necessary to avoid misclassification of very cold clouds and/or land surfaces due to Ch3b's very low signal-to-noise ratio. In addition to the days evaluated in the “round robin” comparison, we selected 12 further training days in 2008 that contain collocations between NOAA-18 and CALIOP, represent COD seasonality, and provide global coverage. Prior to training, all CALIOP COD values > 1 were set to unity. Ancillary data input are the ERA-Interim skin temperature, a snow or ice mask derived from ERA-Interim snow depth and sea-ice concentration, and the USGS land–sea mask. Finally, we applied a simple correction algorithm to remove a cosine viewing-angle dependency of retrieved ANNCOD. This was necessary, as the maximum viewing angle in the AVHRR training dataset was just 35^∘. The binary cloud mask is estimated by classification of ANNCOD data into clear and cloudy through a set of threshold values. The thresholds themselves vary depending on illumination and surface conditions, namely land, sea, and snow or ice cover (Table 2), and were quantified through iterative optimization. They are fixed for all sensors and orbits. As the ANN was trained with AVHRR data only, differences in spectral response functions need to be considered before the ANN can be applied to MODIS and AATSR. We derived appropriate coefficients through linear regression analysis between collocated satellite observations for each input channel pair (Table 3), applying a filter on differences in satellite zenith angle (> 0.5^∘), sun zenith angle (> 1^∘), and observation time (> 30 min). The resulting coefficients were applied to MODIS and AATSR satellite data before ANN input.

We estimate cloud mask uncertainty based on the assumption that this uncertainty is inversely proportional to the difference between retrieved ANNCOD and the threshold applied. As a first step, we generated a CALIOP cloud mask by application of a clear–cloudy threshold value of 0.05. The CALIOP cloud mask is then compared with the collocated ANN mask by quantification of a percent correct (PEC) score. PEC estimates the ratio between all correctly classified pixels and the number of all pixels analysed. Finally, the “truth” uncertainty is defined as 100 − PEC %. We then established the statistical relationship between this uncertainty and the ANNCOD difference to its threshold. Before application of the approach, we normalized differences (ND) to 1. We found a linear correlation between uncertainty and ND for clear cases given by

and a second-order polynomial correlation for cloudy cases (Fig. 2)

The equations of these regression fits are used within CC4CL to quantify cloud mask uncertainty as a function of ND.

Figure 2Neural network cloud mask uncertainty as derived from observations.

Download

3.3.2 Cloud typing

Table 3Linear regression coefficients between collocated AVHRR and MODIS/AATSR channels.

Download Print Version | Download XLSX

Cloud phase is determined by application of the Pavolonis cloud typing algorithm (Pavolonis et al., 2005). The Pavolonis algorithm outputs six cloud types (Table 4), which we then reclassified into water or ice clouds: liquid is fog/warm liquid/supercooled and ice is opaque ice/cirrus/overlap. For CC4CL, the fog type test was deactivated. The algorithm always uses the 0.65, 11, and 12 µm channel data. It reads 3.75 µm data whenever available and 1.65 µm otherwise. These two different approaches produce nearly identical results, except for certain thin clouds and cloud edges (Pavolonis et al., 2005). In addition, we introduced two new cloud types within CC4CL. In response to validation studies, we decided to change the phase of ice clouds whose retrieved CTT is > 273.16 K, the freezing point of water (new cloud type being SWITCHED_TO_ WATER), and of water clouds whose CTT < 233.16 K, the lower limit of supercooled water (SWITCHED_TO_ICE).

The Pavolonis algorithm has weaknesses in detecting cirrus clouds at high latitudes, which are often misclassified as opaque ice clouds. Performance is considerably better when the VIIRS algorithm is used, which provides additional channels and threshold tests. However, these cannot be applied to our AVHRR heritage dataset (Pavolonis et al., 2005).

3.3.3 Optimal estimation retrieval of COT, CER, and CTP

The optimal estimation retrieval ORAC is a non-linear statistical inversion method based on Bayes' theorem (Rodgers, 2009). A state vector containing all variables to be retrieved is optimized to obtain the best fit between observed TOA radiances and radiances simulated by a forward model. The retrieval problem is that of finding the minimum value of a cost function. This function is based on a χ² distribution, which is a combination of the squared deviations between the measurements and the forward model and the retrieved state vector and the a priori state vector, each weighted by their associated uncertainties. The important benefits of ORAC, relative to more traditional retrieval methods, are that cloud parameters are retrieved using information in all satellite channels simultaneously, so that the retrieved parameters provide a robust representation of the shortwave and longwave radiance effects of the observed cloud. The algorithm estimates the retrieval uncertainty, which quantifies the range of values that are feasible considering the uncertainty in the satellite measurements, ancillary data, and ORAC forward model. For a more detailed description of the ORAC algorithm see Part 2 of this publication (McGarragh et al., 2018c).

4.3.1 Case studies

Case study NA1

Study area NA1 is a completely cloud-covered scene over northern Canada containing clear and ice-covered land and open ocean surfaces (Figs. 9, 10). There are a variety of single and multilayered clouds. CC4CL correctly classifies all pixels as cloud-covered, with a few exceptions in sectors 3 and 4 (Figs. 9, 10). CTH retrievals are consistent between the three sensors, only differing in sector 2. CTH is generally lower than CALIOP's top-layer height, unless the latter is optically thick as in sector 4. In the case of a (semi-)transparent cloud top layer, multiple surfaces (several cloud layers, Earth surface) contribute to the observed satellite data. CC4CL CTH is then located closer to, at, or even below the underlying cloud layer (sectors 1, 3, and 2, respectively). For a single-layer cloud with COT > 1, CC4CL and CALIOP CTH agree very well (sector 4). Under such conditions, the retrieval is very accurate. Cloud phase agreement between CC4CL and CALIOP is very variable. It is best for optically thick high ice cloud coverage (sector 1) and worst for low water clouds (sector 4).

Case study NA2

Study area NA2 is located entirely over snow- or ice-free land in western Canada. CALIOP cloud coverage is 4.5 %, spatially broken, and variable in height and phase. Clear-sky pixels are mostly identified by CC4CL (69.3 % correct), and cloudy pixels are occasionally missed (78.7 % correct). CC4CL retrievals of thin high clouds and false positive cloudy pixels have low CTH values (sector 1). Small-scale horizontal variability in CALIOP cloud phase is reflected by CC4CL data, which overestimate the fraction of liquid water clouds in sector 2. CC4CL reproduces CALIOP's spatial variability in CTH, which it slightly underestimates by 0.5–1 km in sector 2. In sector 3 CC4CL considerably underestimates CTH by up to 7 km. Most of these clouds are optically and geometrically thin.

Case study SIB

Study area SIB crosses the Novaya Zemlya islands north of Siberia and is defined by a mixture of open ocean and partially snow- or ice-covered land surfaces. According to both CALIOP and CC4CL, it is completely cloud-covered.

In the event of single-layer cloudiness, CC4CL CTH agrees very well with CALIOP (sector 1 and, in particular, sector 3). The CTH difference between CC4CL retrievals increases in the presence of overlapping clouds (sector 2). There are optically thin but vertically thick (∼ 4 km) clouds in sector 2. For these the retrieved CTH is considerably underestimated by ∼ 6 km, which is probably a result of lower layer contributions that “contaminate” the satellite signals. Overall, about 62.3 % of CC4CL pixels agree with CALIOP phase. Phase mismatch occurs in cases of single-layer optically thin clouds (sector 2) and, less frequently, stratiform cloudiness (sector 3).

Figure 9Study area NA1 (North America 1). Red (Ch1), green (Ch2), and blue (Ch4–Ch5) image derived from NOAA18 data resampled to 0.01^∘ × 0.01^∘ resolution. Date of observation is 22 July 2008, 19:15 LST. Orange lines: extent of the collocated MODIS granule; yellow lines: extent of the collocated AATSR orbit; red line: CALIOP track outside (dashed) and within (solid) study area.

Download

Figure 10Vertical cross section of study area NA1 (North America 1) along the CALIOP track at 5 km horizontal resolution. Top: CTH for CC4CL retrievals (coloured points) and CALIOP measurements (vertical bars), and surface elevation and surface type (blue is open water, green is land, grey is snow/ice). The CALIOP data are shown for those pressure layers where the cumulative top-to-bottom COD exceeds a threshold value of 0 (top layer), 0.15 (mid-layer), and 1 (bottom layer). Bottom: cloud mask/phase (ice to water is red to blue, cloud-free is white, not determined is grey) and type (see Table 4 for key–value pairs) for all three CALIOP layers and CC4CL retrievals. For CC4CL, cloud phase was averaged when resampling, and cloud type was assigned to the most frequent class per grid box. Sectors of characteristic cloud fields are separated by black vertical lines. Number of pixels n = 120.

Download

4.3.2 Case study AFR

Study area AFR is located over the Gulf of Guinea and Western Africa, containing open ocean and snow- and ice-free land surfaces. As we excluded AATSR data, about 10 times more pixel collocations with CALIOP are available (n = 1181) than for previous cases. Additionally, measurements contain tropical and coastal cloud systems such as extensive low-level stratiform cloudiness and continental deep convection (Fig. 15).

In general, the quantitative and qualitative agreement between CC4CL and CALIOP CTH is impressive. CC4CL data track the spatial pattern of continental CTH very well (sector 3), which increases northwards and shows some small-scale variability beyond 8^∘ N. However, CC4CL underestimates CTH of vertically thick clouds and instead places the cloud top at a layer's vertical centre. The height of the stratiform cloud field is almost identical for CC4CL and CALIOP, although CTH of near-surface stratiform clouds is overestimated below thin high cirrus (sector 1). For the small layer located at 4 km height at ∼ 6.7^∘ S and the thin high cirrus layer around 2^∘ S, MODIS retrieval values differ somewhat from CC4CL AVHRR and CALIOP. Again, the phase of optically thick clouds is retrieved very well, but this is not the case for the thin ice cirrus clouds. Generally, CC4CL using the AVHRR heritage channel dataset is almost entirely insensitive to the very high, thin cloud layer in sector 1 (covering stratiform clouds) and 2 (covering the sea surface). Here, CC4CL is rather driven by contributions from very low clouds or the sea surface.

Figure 11Study area NA2 (North America 2). As Fig. 9, but on 22 July 2008, at 20:58 LST.

Download

Figure 12Study area NA2 (North America 2). As Fig. 10, but on 2 July 2008, 20:58 LST (n = 163).

Download

4.3.3 Summary of case studies

The four study areas clearly show that CC4CL retrievals of CTH are very close to CALIOP values for single-layer, optically thick clouds. For significant extents of the regions presented, the CTH is accurate to within 240 m. For multilayer clouds, CC4CL estimates are almost exclusively located in between CALIOP's top and bottom layer estimates. For these cases, the optimal estimation algorithm processes satellite signals that are likely to contain radiance contributions from multiple cloud layers. The OE then optimizes the fit between modelled and observed radiances by placing the cloud lower in the atmospheric profile, and so the mixed nature of the satellite data leads to an underestimation of CTH. The results shown here are a representative sample from an extensive validation performed within the Cloud_cci project (Stengel et al., 2017).

There is no clear influence of the underlying land type or topography on retrieval values or the cloud mask. However, the limited sample size does not allow for generalizations. For site NA2, CALIOP identified cloud-free pixels, 69.3 % of which were also detected as cloud-free by CC4CL's neural network cloud mask, and with few exceptions as low-level water clouds otherwise. In relatively few cases, CC4CL fails to detect clouds seen by CALIOP (% of missed clouds is 9.0 (NA1), 21.3 (NA2), 0.6 (SIB), 3.1 (AFR)). We did not account for fractional cloud coverage, as we set a grid box as cloud-covered when any corresponding CC4CL pixel contains cloud information. As a consequence, there are slightly more cloud-covered pixels for the spatially higher resolved MODIS and AATSR data than for AVHRR.

The CC4CL phase identification does not agree with any of the three CALIOP cloud flags consistently, which is reasonable given the differences between active and passive observations. After rounding CC4CL values to the nearest integer, the percentage of pixels with equal phase is lowest for the top layer at COD > 0 (NA1 = 49.9 %, NA2 = 32.2 %, SIB = 62.3 %, AFR = 53.0 %), but similar for the mid-layer COD > 0.15 (NA1 = 53.1 %, NA2 = 46.5 %, SIB = 66.4 %, AFR = 94.7 %) and the bottom layer COD > 1 (NA1 = 47.3 %, NA2 = 57.4 %, SIB = 66.3 %, AFR = 91.1 %). These values do show, however, that phase determination performs very well when optically thick clouds dominate, as is the case for study area AFR. When averaged over all layers, phase agreement is largest for site AFR (79.6 %), followed by SIB (65.0 %), and clearly lower for NA1 (50.1 %) and NA2 (45.4 %).

For ice clouds, the most frequently occurring cloud types are cirrus (ID = 6) for CALIOP and overlap (ID = 8) or cirrus (ID = 7) for CC4CL. Water cloud types are more heterogeneous and for CALIOP predominantly low transparent (ID = 0), but altostratus (ID = 5) and altocumulus (ID = 4) are also frequent. CC4CL water clouds are approximately equally distributed amongst water (ID = 3) and supercooled (ID = 4) cloud types.

Figure 13Study area SIB (Siberia). As Fig. 9, but on 27 July 2008, 08:10 LST.

Download

Figure 14Study area SIB (Siberia). As Fig. 10, but on 27 July 2008, 08:10 LST (n = 116).

Download

The scenes investigated and discussed here are just a small subset of the large variety of global cloudiness. We could only find collocations with AATSR data at high latitudes, where multilayer cloud coverage is common. Under such difficult conditions, the retrieved CTH is a mixture of all radiatively contributing cloud layers. Please refer to Stengel et al. (2017) for a quantitative, global validation of CC4CL cloud properties.

Figure 15Study area AFR (Africa). As Fig. 9, but on 24 October 2009, 13:45 LST.

Download

Figure 16Study area AFR (Africa). As Fig. 10, but on 24 October 2009, 13:45 LST. Due to space restrictions, no cloud type values are shown in table. n = 1181.

Download