Top-ofthe-atmosphere shortwave flux estimation from satellite observations : an empirical neural network approach applied with data from the A-train constellation

Estimates of top-of-the-atmosphere (TOA) radiative flux are essential for the understanding of Earth’s energy budget and climate system. Clouds, aerosols, water vapor, and ozone (O3) are among the most important atmospheric agents impacting the Earth’s shortwave (SW) radiation budget. There are several sensors in orbit that provide independent information related to these parameters. Having coincident information from these sensors is important for understanding their potential contributions. The A-train constellation of satellites provides a unique opportunity to analyze data from several of these sensors. In this paper, retrievals of cloud/aerosol parameters and total column ozone (TCO) from the Aura Ozone Monitoring Instrument (OMI) have been collocated with the Aqua Clouds and Earth’s Radiant Energy System (CERES) estimates of total reflected TOA outgoing SW flux (SWF). We use these data to develop a variety of neural networks that estimate TOA SWF globally over ocean and land using only OMI data and other ancillary information as inputs and CERES TOA SWF as the output for training purposes. OMI-estimated TOA SWF from the trained neural networks reproduces independent CERES data with high fidelity. The global mean daily TOA SWF calculated from OMI is consistently within ±1 % of CERES throughout the year 2007. Application of our neural network method to other sensors that provide similar retrieved parameters, both past and future, can produce similar estimates TOA SWF. For example, the well-calibrated Total Ozone Mapping Spectrometer (TOMS) series could provide estimates of TOA SWF dating back to late 1978.


Introduction
The Earth's energy budget constrains the general circulation of the atmosphere and determines the climate of the Earthatmosphere system; it is therefore also an indicator of possible climate changes (Hatzianastassiou, et al., 2004).There is a long history of attempts to estimate Earth's albedo and energy budget (Dines, 1917;Hartmann et al., 1986).With the advent of the satellite remote sensing era, it became possible to directly measure the albedo of the Earth.Subsequently, the shortwave (SW) energy balance at the top of the atmosphere (TOA) and the role of clouds, aerosols, and trace gases has been studied using satellite measurements (Ramanathan et al., 1989;Yu et al., 2006;Bellouin et al., 2005;Loeb et al., 2005;Patadia et al., 2008;Joiner et al., 2009).
The Earth Radiation Budget Experiment (ERBE) was launched in October 1984 by the space shuttle Challenger and provided long-and shortwave radiation parameter measurements.TOA SW radiative parameter estimates from ERBE (Barkstrom, 1984;Barkstrom and Smith, 1986) showed that clouds approximately double the albedo of Earth to an all-sky value of 0.3 from an estimated clear-sky value of 0.15 (Ramanathan et al., 1989;Harrison et al., 1990).The next generation of broadband instruments, the Cloud and the Earth's Radiant Energy System (CERES), draws heavily on ERBE heritage.Since its first launch in 1997 on board the NASA Tropical Rainfall Measurement Mission (TRMM), CERES has provided continuous observations that can be used to understand the role of clouds and the energy cycle in global climate change (Wielicki et al., 1995;Loeb et al., 2012).
Continuous and coincident measurements of radiative fluxes and atmospheric components facilitate research studies to estimate and understand the role of different atmospheric components on the planetary energy balance.Although CERES provides state-of-the-art estimates of TOA radiative fluxes, it was not designed to make measurements of individual atmospheric components that impact those fluxes.Several studies have utilized aerosol and cloud information from high spatial resolution MODerate resolution Imaging Spectroradiometer (MODIS) measurements to quantify their impact on TOA fluxes (Yu et al., 2006;Patadia et al., 2008;Zhang et al., 2005b;Loeb et al., 2005;Oreopoulos et al., 2009).Several attempts have also been made to convert narrowband radiances into broadband fluxes using regression or more sophisticated statistical approaches (Chevallier et al., 1998;Hu et al., 2002;Domenech and Wehr, 2011;Vázquez-Navarro et al., 2013).
The Ozone Monitoring Instrument (OMI), flying on NASA's Aura satellite since 2004, provides information about components important for the Earth's SW radiation budget, including the effective cloud/aerosol fraction (Stammes et al., 2008;Joiner and Vasilkov, 2006) and total column ozone (TCO) (Veefkind et al., 2006;McPeters et al., 2008;Kroon et al., 2008).OMI-retrieved parameters can be utilized to understand their role in the Earth's SW energy budget.
Modeling the spatial and temporal distribution of the TOA shortwave flux (SWF) requires a description of the components that control the transfer of solar radiation within the Earth-atmosphere system.When required parameters are missing or incomplete, a statistical approach is an alternative for estimation of TOA SWF.Here, we develop an artificial neural network model (NNM) to estimate total reflected TOA outgoing SWF.Artificial NNs are algorithms that simulate biological NNs by learning and pattern recognition (Bishop, 1995).NNs have been used by many scientific disciplines, including Earth science, to identify patterns and extract trends in imprecise and complicated nonlinear data (e.g., Lee et al., 1990;Gupta and Christopher, 2009).In radiation studies, NNs have been used to estimate TOA and surface SWF based on radiative transfer calculations with or without data from satellites (e.g., Krasnopolsky et al., 2008Krasnopolsky et al., , 2010;;Takenaka et al., 2011;Vázquez-Navarro et al., 2012;Jiang et al., 2014).CERES TOA flux algorithms have also used NNs to generate angular distribution models (ADMs) in the absence of sufficient high-resolution imager information for reliable scene identification (Loukachine and Loeb, 2003Loeb, , 2004)).
In this study, we utilize OMI cloud and ozone products along with other ancillary data to estimate TOA SWF.We develop NNs that take OMI-derived quantities as inputs and provide CERES-equivalent TOA SWF as the output.In essence, the trained NNs have learned or incorporated all of the complexity that is essential to the CERES algorithms (ADMs, scene identification, etc.) and are then able to predict TOA SWF directly and efficiently based on a limited number of retrieved products from OMI or sensors that provide similar data.The trained NN models are optimized to run with data sets from OMI or similar sensors and can be applied generally to different seasons and years.For example, the NN-based models we develop here can be applied to similar measurements from the Total Ozone Mapping Spectrometer (TOMS) instruments.One objective of this study is to assess how well TOA SWF can be estimated using OMI-derived cloud and ozone products with NNs when nearly coincident CERES data are used for training.The developed NNs can then be applied to other data sets with similar products and accuracy.Alternatively, the general training approach could be applied with similar data sets such as with MODIS cloud and ozone products.
The strength of a NN approach is that it is highly efficient and, if well-trained, should be precise and accurate for this type of problem.NNs may be used to examine the sensitivity of TOA SWF to various input data sets but do not themselves provide specific insight into the physical mechanisms behind those sensitivities.NNs will of course only be as good as the data that are used for training.In addition, they may not perform well for unusual conditions that are not present in the training data set.Therefore, NNs cannot replace dedicated TOA SWF estimates from instruments like CERES.
The paper is organized as follows: Sect. 2 describes the various satellite data sets utilized in the study.Section 3 discusses the development of NN models including the selection of input parameters.Section 4 evaluates our NN estimation of TOA SWF using independent CERES data over ocean and land.Section 5 summarizes the results and discusses future work.

Satellite data sets and coincident sampling
Under clear-sky conditions, TOA SWF is affected by the Earth's surface properties, atmospheric absorbers such as water vapor, ozone, and aerosols, and scattering by air molecules and particulates.Over ocean, surface properties can be characterized by ocean color and roughness of the ocean surface.Under cloudy sky conditions, cloud optical properties such as the cloud optical thickness, geometrical cloud fraction, effective radius, and phase function affect TOA SWF.In clear and cloudy skies, the solar zenith angle (SZA) and Sun-Earth distance (SED) impact the TOA SWF.
In this work, we make use of data sets mainly from two passive sensors in A-train constellation of satellites that fly within 15 min of each other: (1) Aura OMI with an equatorial crossing time of ∼ 13:45 ±15 min LT (local time) and (2) Aqua CERES with an equatorial crossing time of ∼ 13:30 LT.For testing and optimization of the NNMs, we primarily use 2007 data over global oceans.Starting around 2008, OMI experienced an anomaly presumably due to mate-rial outside the sensor that adversely affects the quality of the level 1B and level 2 data products in a portion of its 60 rows across the swath.Our study focuses on data in 2007 that are not significantly affected by these anomalies.

CERES
The first CERES instrument flew on the TRMM satellite, launched in November 1997, and provided data until 2000.Five CERES instruments are currently operating: two on NASA's Terra satellite (FM1 and FM2), two on NASA's Aqua satellite (FM3 and FM4), and one on the Suomi National Polar-orbiting Partnership (NPP) satellite (FM5).These CERES instruments provide radiometric measurements of the Earth's atmosphere from three broadband channels: (1) a shortwave channel to measure reflected sunlight (0.3-5 µm), (2) a long-wave channel to measure Earthemitted thermal radiation in the window region (8-12 µm), and (3) a total channel to measure radiation from 0.3 to 200 µm.
CERES radiances are converted to TOA fluxes using ADMs.The CERES science team has an extensive database of ADMs for clear-and cloudy-sky over both land and ocean (Loeb et al., 2005).The ADMs heavily depend upon the observed scene type and are sensitive to surface characteristics as well as cloud and aerosol optical properties (Loeb et al., 2003;Zhang et al., 2005a;Patadia et al., 2011).The ADMs over ocean are dependent upon wind speed and aerosol optical thickness along with sun-satellite geometry (Zhang et al., 2005a).
The Aqua spacecraft carries two identical CERES instruments: one operates in a cross-track scan mode (FM3) and the other in a biaxial scan mode (FM4).Measurements from the biaxial scan mode were used to develop the ADMs; this provided considerable improvement over the previous generation of instruments, including the ERBE (Loeb et al., 2003(Loeb et al., , 2007)).
This study uses the Single Scanner Footprint (SSF, Edition 3A) TOA SWF obtained from the Aqua CERES FM3.The SSF product is an instantaneous merge of CERES parameters with coincident cloud and aerosol parameters derived from the Aqua MODIS (Loeb et al., 2003) at the footprint level (i.e., not daily averages).The high-resolution (1 × 1 km 2 at nadir) MODIS imager data are used to characterize the clear and cloudy portions of the larger CERES pixel (20 × 20 km 2 at nadir).We also used broadband surface albedo over land derived using measurements from MODIS and CERES to characterize the land surface in the NN trained specifically over land.

OMI
OMI provides hyper-spectral measurements of Earthbackscattered sunlight from UV to visible wavelengths (∼ 270-500 nm) with a spectral resolution of the order of 0.5 nm (Levelt et al., 2006).Its spatial resolution is 13 × 24 km 2 at nadir with a swath width of about 2600 km.Cloud, aerosol, and TCO products from OMI are used in this study.Specifically, the cloud-aerosol optical centroid pressure (OCP), effective cloud fraction (ECF; f c ), Lambertian equivalent reflectivity (LER) at 354.1 nm, SZA, relative azimuth angle (RAA), and viewing zenith angle (VZA) are obtained from the OMI cloud products as detailed below, and aerosol index (AI) and TCO are obtained from the OMI-TOMS total ozone product (OMTO3, version 8.5, collection 3) (McPeters et al., 2008).
Cloud-aerosol OCP, also known as effective cloud pressure, is a measure of the reflectance-weighted pressure reached by incoming solar photons (Joiner et al., 2012).It is distinct from the cloud-top pressure (CTP).While CTP is the more important parameter needed for TOA long-wave flux, OCP is more related to atmospheric absorption in the shortwave.OCP is derived from OMI observations using two different methods (Stammes et al., 2008): (1) filling-in of solar Fraunhofer lines from rotational Raman scattering in the UV (the OMCLDRR product) (Joiner and Bhartia, 1995;Joiner et al., 2004) and ( 2) collision-induced oxygen absorption (O 2 -O 2 ) at 477 nm (the OMCLDO2 product) (Stammes et al., 2008;Acarreta et al., 2004).Unless otherwise specified, we use the f c and OCP from OMCLDRR product here.
OMI cloud and trace-gas algorithms use a simplified mixed Lambertian cloud model to estimate observed radiances I m .In this scenario, a pixel is modeled as a having components from clear and cloudy sub-pixels weighted using an ECF f c ; i.e., where I g and I c are the radiances computed in the Rayleigh atmosphere for Lambertian surfaces corresponding to the clear and cloudy portions of the scene, respectively; f c is defined as the fraction of the Lambertian cloud covering the pixel and is related to both the geometric cloud fraction and cloud optical thickness.It contains information similar to the LER of the scene (related to cloud and surface reflectivities).However, because it attempts to account for variations in the Earth's surface reflectivity, it is a more spectrally invariant quantity and therefore potentially more highly correlated with TOA SWF.Formally, the ECF is wavelength dependent because it is defined by spectral quantities (Stammes et al., 2008).We conducted a simulation experiment to evaluate the wavelength dependence of f c .In this experiment, we simulate observed TOA radiances as a weighted sum of the clear-sky and cloudy radiances; i.e., where I c is the cloudy radiance computed with a planeparallel cloud model that depends on cloud optical thickness, and f g is the geometrical cloud fraction.In our simulation, clouds have a vertically uniform distribution of the www.atmos-meas-tech.net/9/2813/2016/extinction coefficient and phase function.We use a cloudtop height of 5 km and a cloud layer thickness of 1 km.The assumed cloud optical depth (COD) of 20 is spectrally independent within the 320-1400 nm wavelength range.The spectrally independent optical thickness is a good approximation for clouds with sufficiently large particles (Deirmendjian, 1969).We neglect gaseous absorption in the specified spectral range.Three models of cloud phase function are used: (1) ice crystals with an effective diameter of 60 µm (Baum et al., 2014), (2) Deirmendjian's C1 model of a water cloud having droplets with an effective diameter of 12 µm (Deirmendjian, 1969), and (3) the Henyey-Greenstein model (e.g., van de Hulst and Irvine, 1963) with an asymmetry parameter of 0.85.We use a simplified model of the spectral ground reflectance: R g = 0.05 at λ < 700 µm and R g = 0.2 at λ > 700 µm.We then calculate f c by inverting Eq. ( 1) assuming a Lambertian cloud with a reflectivity of 0.8; this is commonly used for trace-gas algorithms (Stammes et al., 2008).If f c does not vary much with wavelength, then it should be highly correlated with TOA SWF and thus a good predictor in a statistical model of TOA SWF.We investigate the spectral dependence of f c in Fig. 1 where we performed calculations using f g = 0.5, SZA = 45 • , and at nadir for the three phase functions assuming a cloud single-scattering albedo of unity.It can be seen that f c is nearly invariant with wavelength over a wide spectral range; it changes by only a few percent even for the steep change in the ground reflectance (simulating the so-called red edge) at 700 nm.COD of less than 20 at the same geometrical cloud fraction leads to a lower value of the ECF.However, the spectral variations of ECF remain similar to that shown in Fig. 1.The weak spectral dependence of ECF is explained by the fact that the Lambertian cloud model with cloud reflectivity of 80 % effectively accounts for Rayleigh scattering in partially cloudy scenes as it has been shown by Ahmad et al. (2004).This result holds when other input parameters in our simulation are varied.We will show the implications of the spectral invariance of f c below.
We have used the following modified cloud fraction parameter, f c_mod , as a predictor to estimate TOA SWF: where SED is the Sun-Earth distance.The modification accounts for variation in the incoming solar irradiance with respect to SZA and SED. Figure 2 shows a 2-D histogram for a month of collocated CERES TOA SWF and OMI f c_mod (collocation criteria described in detail below) that demonstrates the near-linear relationship between the two parameters.It also indicates that the single parameter f c_mod captures much of the variability in TOA SWF.

Ancillary data
In addition to OMI data, a SeaWiFs-derived chlorophyll (Chl) concentration climatology is used as an input predic-  tor when f c < 1.The precipitable water (PW) and 2 m surface wind speed (wind) are also used as predictors; these are provided in the CERES SSF data set and are taken from the Goddard Earth Observing System (GEOS) 4 reanalysis (Bloom et al., 2005).

Coincident sampling of OMI and CERES
Because the sizes of the OMI (13 km × 24 km) and CERES (20 km 2 ) pixels are similar at nadir, we perform a simple spatial collocation by finding the closest CERES pixel corresponding to each OMI pixel.OMI and CERES collocated pixels are only included in our training and validation samples when the distance between centers of OMI and CERES pixels is less than 20 km.We do not include pixels with viewing zenith angles > 60 • .At these angles, OMI and CERES pixels become significantly larger (∼ 150 km for OMI and ∼ 200 km for CERES in the cross track direction) and therefore may contain many different scene types.We also mask OMI pixels with AI > 1 to avoid heavy absorbing aerosol loaded scenes where the f c and OCP are known to contain errors (Vasilkov et al., 2008).This does remove not nonabsorbing aerosols.Non-absorbing aerosols are essentially treated as clouds within the OMI cloud data sets.We examined the frequency distribution of the distance between OMI and CERES pixels of all the collocated data sets and found that most of the collocated data, 98 and 60 % have distances less than 20 and 10 km, respectively.The quality-controlled collocated data are then averaged on equal latitude and longitude grids of 1 • × 1 • (unless otherwise specified) for training, testing, and validation of the neural networks.
The NN inputs (predominantly retrievals from OMI measurements) are used to train the NN to match the output (CERES-derived TOA SWF).Once the network is trained, input data sets can be used to calculate TOA SWF with characteristics similar to the CERES product.Therefore, the NNproduced TOA SWF will be referred as OMI estimated SWF throughout the paper.
3 Artificial neural network model

General NN architecture and training approach
The general neural network architecture has three layers of neurons: an input layer, a hidden layer, and an output layer with standard multi-layer network architecture.We use the same number of neurons in the hidden layer as in the input layer as this produced generally good results.The input layer has an identity activation function; all other layers are connected by sigmoid activation functions (Eq.4).
The network normalizes both input and output data sets with a unique linear mapping for each input and output parameter.Figure 3 provides an example of a schematic of the network used in our study.Here we used two different NNMs: one with nine nodes or parameters (NNM1) and a second with seven nodes (NNM2) in the input layer.Both of these models have one node (TOA SWF) in the output layer.Figure 3 also lists the input parameters corresponding to the NNM1 and NNM2 models.The NNM1 model is optimized for ocean cases where the OMI f c < 1.0, whereas NNM2 is optimized for cases where f c = 1 (saturated cases).
NNMs require optimized training to produce accurate outputs.Here we use a standard back propagation training algorithm (Hertz et al., 1991), where inputs are iteratively sent to the neural network.In back propagation, the hidden layer weights associated with each input parameter are modified through the training process that minimizes errors between the targets and outputs (Bishop, 1995;Gardner and Dorling, 1998).After each iteration, the error is propagated backward through the network and weights are modified to bring the actual response of the network closer to the desired output in a statistical sense.The function minimized during the training is a sum of squared errors of each output for each training pattern.Once the network is trained, it can be evaluated using independent data (i.e., not used in the training data set).

Impact of different input parameters
Here we examine the impact of using various input parameters on the derived neural networks.This exercise is performed using data with f c < 1 with 1 month of the data over ocean (January 2007).Table 1 presents the performance of eight different NNMs, denoted models "a" through "h", with various input parameters listed in Table 1 and described in more detail in Sect.3.2.1-3.2.2.

Inclusion of OMI UV-derived parameters
In model a, we have combined the effects of SZA, SED, and f c into a single input parameter called f c_mod , which was defined in Sect.2.2.Use of this modified input parameter alone explains about 94 % (R = 0.97) of the variabil- ity in TOA SWF.As we add other parameters in models b through h, we observe small improvements in the OMIestimated TOA SWF. Figure 4 shows the spatial distribution of monthly mean OMI-CERES SWF differences for these models.
The TOA SWF is estimated from a measured radiance and therefore the observational geometry factors in.The addition of satellite-viewing geometry parameters (VZA, RAA) to model a provides improvements in areas of high biases and reduces the standard deviation from 37.1 to 31.4 Wm −2 .Model c tests the ability of the 354 nm reflectivity (LER) to predict TOA SWF in place of f c_mod .Although the statistical parameters in Table 1 corresponding to models b and c are very similar, we note spatial differences in the OMI-CERES TOA SWF in Fig. 4. Further analysis reveals that the f c -based model b provides more accurate flux estimation as compared with the LER-based model for a larger range of fluxes.
The inclusion of TCO (model d) as an input parameter positively impacts TOA SWF estimation as shown in Fig. 4d; the high positive biases in the tropical Pacific and Indian oceans and in the region near 60 • N have been reduced.The percentage of monthly mean OMI-CERES data that fall within ±8 % increases from 91 % in model c to 94 % in model d.
Model e adds cloud OCP to the input parameters included in model b.OCP also improves SWF estimates; the regions where improvement occurs are different from those improved by using TCO.Model f shows that when TCO and OCP are used together as input parameters, there is further improvement in SWF estimation.Although the global statistics in Table 1 do not clearly reflect this improvement, Fig. 4f shows that inclusion of OCP and TCO reduces biases in many regions, most prominently in the tropics.The percentage of total OMI samples (monthly mean) within ±8 % of CERES increases from 92 % in model b to 95 % in model f.Apart from these parameters, we also evaluated the inclusion of AI as an input parameter (not shown here).We found that overall it does not significantly improve the results; however it does provide some improvement in regions with positive AI values.

Addition of meteorological and other ancillary data
The impact of surface winds and total column water vapor (model g in Fig. 4g) is more prominent in the tropics than in other regions.Inclusion of Chl and LERs in model h removes some of the notable low biases in TOA SWF near the coast of northern China, the Caspian Sea, and the Black Sea.Further- more, model h corrects for negative biases in areas with high TOA SWFs, most likely due to the inclusion of LER.The model "h" produces 89 % (99 %) of OMI-estimated monthly mean TOA SWFs within ±5 % (±12 %) of CERES and is the best of the eight models.In all of the results, some striped patterns appear in the difference maps.These stripes are also seen in difference maps for July and will be discussed below in Sect.4.3.

Consistency over time
We next examine the performance of the NN model h with respect to different input samples.We first examine the robustness of the NN for detection of interannual variability.
In this exercise, we trained the NN with data from the first 15 days of January 2007 as above and applied it to input data from the entire months of January 2007 (Fig. 5a) and January 2006 (Fig. 5b). Figure 5a-b present 2-D histograms similar to that in Fig. 2 but here compare the TOA SWF from the NN with that from CERES for January of the 2 different years over ocean.The colors represent the 2-D histogram (or density) of coincident pairs using a bin size of 10 Wm −2 .The solid black 1 to 1 line is shown with three dotted lines on both sides that represent envelopes of ±5, ±10, and ±15 % OMI-CERES differences.Figure 5a-b show that the NN performance is consistent between years.Although the number of samples in January 2006 and 2007 is a bit different, the NN model produces similar statistics.
A NN trained on 1 particular month of data is not guaranteed to perform well for a different month.In the next test, we used the NN model trained on January 2007 data with input data from July 2007.Results for July (using CERES as the benchmark) were degraded as compared with application to January data.This is due presumably to changes in observing conditions between the 2 different months (changes in solar angles).
We then trained the same NN (identical input parameters) using a subset of data from July 2007 and applied it to data from the entire month of July 2007.Results (again using CERES as the benchmark) were of similar quality to those where the NN was trained and applied to January.This exercise suggests that we may need to use different models for different months or expand our training data set for application to different months.
We next use data from the 1st day of each month of 2007 for training and data from the 16th day of each month of 2007 for evaluation.The comparisons with CERES using the training and validation data are consistent as shown in Fig. 6.
The almost identical values of statistical parameters for training and validation data demonstrate that the neural network has been well trained.For example, there is a high degree of linear correlation (R = 0.98 or R 2 = 0.96) and slopes close to 1 (0.96) in both training and validation comparisons.The mean bias in both training and validation data sets is close to zero, whereas the global standard deviation remains stable and close to 30 Wm −2 in the two independent model runs.Further analysis shows that 83, 70, and 43 % of TOA SWF estimated from OMI (training and validation data combine) lie within the 15, 10, and 5 % of the CERES TOA SWF, respectively.
Further evaluation of the entire year reveals that this NN (model h) is appropriate for all months.Therefore this model will be used for subsequent analyses in this study.Creating more networks as a function of scene type or for different latitude belts or even for different months/seasons may improve results in certain regions.However, based on our results, we simplified the approach by minimizing the number of networks.

Case of f c = 1
About 1-2 % of total coincident data correspond to f c = 1, typical of overcast conditions with optically thick clouds.These cases were modeled using a simpler NN model with inputs of LER, SZA, VZA, RAA, OCP, O 3 , and PW; the surface-related parameters (surface wind speed and Chl content) do not produce a significant impact for ECF = 1 and have therefore been removed.Subsequent all-sky results shown in Fig. 6 use combined outputs from the two separate models for f c < 1 and f c = 1.     2. Model "h" is used to produce these results.The three dotted lines are 5, 10, and 15 % error envelope lines.sented in Fig. 6.The RMSE varies between about 24 and 35 Wm −2 and continuously increases with cloud fraction (and observed TOA SWF).The NRMSE, in contrast, continuously decreases with ECF from about 18 % for 5 % ECF to ∼ 6 % for overcast conditions.The bias represents the mean error (in percent) for each ECF bin.The mean global bias shows more variability than RMSE and is highest (2.9 %) for about 10 % ECF.The bias decreases sharply from 2.9 % at f c = 0.1 to about 1.2 % at f c = 0.4.The bias remains low (< 1.2 %) for f c > 0.4 (usually associated with frontal or deep convective clouds).The higher biases for lower f c (usually associated with thin cirrus and broken clouds) are likely related to higher noise and uncertainties in OMI cloud parameters.For example, Joiner et al. (2012) showed that cloud OCP errors increase with decreasing f c .The biases may also be related to absorbing aerosol in the scene, particularly when it overlies clouds.This will be illustrated in more detail below as we show spatial variations in OMI-CERES differences.
The corresponding statistical parameters are listed in Table 2.The three dotted lines are 5, 10, and 15 % error envelope lines.

Effects of spatial and temporal averaging
In order to evaluate the NN performance at different spatial and temporal scales similar to those used by the climate community, we use data from July 2007.Figure 8 presents a comparison of daily CERES and OMI TOA SWF over ocean for six spatial scales: the OMI native pixel (13 × 24 km 2 at nadir) and 0.5, 1, 2, 5, and 10 • gridded spatial resolutions.Statistical parameters for these comparisons are reported in Table 2.As expected, the pixel level data are much noisier than the gridded data due to collocation noise in partly cloudy cases.This collocation noise averages out over larger spatial and temporal scales.Regardless of the noise, the slope (0.96) is still close to 1, and the linear correlation coefficient is 0.96 with a standard deviation of 47.7 Wm −2 .Below 300 Wm −2 , where the sample density is highest, the NN slightly underestimates the CERES SWF.The mean bias of the OMIestimated SWF with respect to CERES is −1.4 Wm −2 .This bias may be due to a combination of effects including uncertainties in the input parameters as well as the limitations of the NN model itself.For example, we have excluded pixels with a clear signature of absorbing aerosols (OMI-derived AI > 1) where OMI ECFs and pressures may be in error in both the training and validation data.However, in some regions where smoke and dust overlaying clouds is common (e.g., western coast of Africa), pixels with erroneous cloud fractions due to small amounts of absorbing aerosol may be present in both the training and validation data.This may produce errors in the NN model and will be examined in more detail below.For the daily data, as the spatial averaging scales increase from 0.5 to 10 • , the OMI-estimated SWF becomes almost identical to CERES; the correlation coefficient in- Note: N is number of pairs, R is the correlation coefficient, M the slope, I is the intercept, bias is the mean of (OMI-CERES in Wm −2 ), SD is standard deviation of (OMI-CERES) in Wm −2 , EE is the error envelope for 5, 10, and 15 % errors.All flux values have units of Wm −2 .creases from 0.97 to 0.99, and the slope increases from 0.96 to 0.97.The percentage of OMI data that fall within ±5 % of CERES increases from 37 % for 0.5 • to 69 % for 10 • grids.About 87 % percent of OMI-estimated 2 • gridded daily mean TOA SWFs are within 15 % of CERES data.Figure 9a-d show 2-D histograms of monthly mean gridded data over ocean at 0.5, 1, 2, and 5 • spatial resolutions, respectively.The monthly intercomparisons of OMI and CERES SWF show excellent agreement at all spatial resolutions with correlation coefficients of 0.99 and slopes of 0.98 (Table 2).The global mean biases vary between −1.8 and 0.25 Wm −2 .The standard deviations vary between 6.6 and 12.9 Wm −2 for the different spatial resolutions.Of monthly mean 1 • gridded OMI estimated TOA SWFs, 97 % are within 15 % of those derived from CERES and 93 % are within 10 %.

Spatial distribution of TOA SWFs over ocean
Figure 10 presents the spatial distribution of 1 • gridded monthly mean (July 2007) TOA SWF from CERES (Fig. 10a) and the difference with the OMI in Wm −2 (Fig. 10b) and percent difference (Fig. 10c).There are subtle differences between the NN and CERES estimates of TOA SWF as shown in Fig. 10b and c.The OMI minus CERES histograms (Fig. 10d) show that for 44 % (79 %) samples, NN fluxes are within ±2 (±5) % of CERES fluxes.About 9 % of the samples have biases of ±8 % or more.Overall, the Northern Hemisphere shows better agreement than the Southern Hemisphere during July (boreal summer).This could be due to larger errors in the OMI cloud products at higher SZAs in the Southern Hemisphere.The low biases on the western coast of Africa may be due to the presence of absorbing aerosols, particularly when they occur over clouds.The striped pattern in the Southern Hemisphere (latitudes > 40 • S) is mainly associated with high VZAs in conjunction with high SZAs that occur on one side of the OMI swath.
Figure 11 similarly shows differences between CERES and OMI TOA SWF over ocean derived using f c and OCP from the OMI O 2 -O 2 product in place of the OMI RRS product.Because there are slight differences in the two cloud products, we retrained the network with OMI O 2 -O 2 cloud parameters to be consistent.The use of the O 2 -O 2 f c and OCP improves the accuracy of the estimated TOA SWF.The regions of improvement include the western coasts of South America and southern Africa and some parts of the Indian Ocean.The O 2 -O 2 cloud product, which uses visible wavelengths, is less affected by absorbing aerosol; this may explain the improvement in these areas where absorbing aerosol, especially over clouds, is common.However, negative biases remain over large regions off the western coast of Africa.
Figure 12a shows a time series of daily global mean values of TOA SWF over ocean from OMI and CERES for 2007.Both instruments show almost identical daily variations with differences within ±1 %. Figure 12b and c provide monthly averaged (July 2007) CERES and OMI zonal and meridional means of TOA SWF.The OMI-derived TOA SWF is able to reproduce the variability shown in the CERES data well.

Spatial distribution of TOA SWFs over land
We developed a similar land-only NN model that utilizes most of the input parameters from our ocean NN (e.g., OMI RRS cloud parameters).The only change is that for surface characterization we use a monthly climatology of surface broadband albedo in place of the Chl concentration and surface wind speed.The albedo product is derived using a combination of CERES and MODIS observations at 1 • spatial resolution (Rutan et al., 2009).The land NN model is trained and validated over land only and it follows a similar approach as was used over ocean but with a slightly different set of input parameters.We have not performed extended analyses and testing over land as we have over ocean.Here, we demonstrate that the same methodology with different input parameters can reproduce CERES TOA SWF over land with similar accuracy and precision as was obtained over ocean.
Figure 13 shows results from the OMI-derived CEREStrained NN that produces TOA SWF over land.Statistical comparison with CERES over land provides results similar to those over ocean.The NN performs well over Asia and parts of Europe and the Americas.The OMI-based NN tends to underestimate TOA SWF over the high albedo desert areas of Northern Africa, Australia, and also over some regions of South America.Note that the large differences that occur in coastal regions may be due to imperfect collocations.

Summary and conclusions
We have developed a neural network approach to estimate TOA SWF based primarily on UV parameters retrieved with the Aura OMI and Aqua CERES-derived TOA SWF used for training.One year of data from OMI and CERES has been used to train/validate/analyze several separate neural networks for different conditions, which together provide estimation of TOA SWF under all-sky conditions.The most important input parameters are ECF and sun-satellite geometry.TCO and cloud optical centroid pressure from OMI, as well as surface-related parameters, provide secondary positive impacts.
Independent validation at different spatial and temporal scales shows that the OMI NN-based approach reproduces CERES-derived TOA SWF with high fidelity.Correlation coefficients for all comparison are > 0.95, and slopes are close to unity.A high percentage of OMI-estimated monthly mean TOA SWF at 0.5 • spatial resolution over global oceans (97 %) falls within 15 % of CERES.The global mean bias in pixel level data of about −1.4 Wm −2 over oceans with respect to CERES is likely due in part to errors in OMI cloud parameters that occur in the presence of absorbing aerosols.
We plan to apply our derived neural networks to longterm, well-calibrated UV measurements from TOMS.The TOMS series provides a long-term data record dating back to late 1978 (about half a decade before the first ERBE launch) P. Gupta et al.: Shortwave flux estimation from OMI observations with a few small gaps between that time and the first CERES launch.We should be able to apply NN models derived with CERES/OMI to TOMS, provided that the input parameters are either available and compatible or can be estimated independently.For example, in place of actual cloud OCPs that are available from OMI, but not from TOMS, we could use a cloud OCP climatology that was developed from OMI data for use in the TOMS total ozone algorithm.The lower spatial resolution of TOMS is not expected to present any difficulties.This approach can also be extended to the future geostationary missions that provide the relevant input data, such as the NASA Earth Ventures Tropospheric Emissions: Monitoring of Pollution (TEMPO), the Korean Geostationary Environmental Monitoring Spectrometer (GEMS), and the European Space Agency (ESA) Sentinel 4 (Al-Saadi et al., 2015).Finally, we may apply the NN training and evaluation approach to data from CERES and the nadir mapper on the Ozone Mapping Profiling Suite (OMPS) that provides information similar to OMI.Both instruments fly on the Suomi NPP satellite.This may reduce collocation noise and small biases that result from the time difference between OMI and CERES measurements.The final NN models developed in this study (e.g., NNM-1 and NNM-2 in Fig. 3) along with the instructions on how to use them have been provided in the Supplement.
The Supplement related to this article is available online at doi:10.5194/amt-9-2813-2016-supplement.

Figure 2 .
Figure 2. Two-dimensional histogram of effective cloud fraction (ECF or f c ) normalized with respect to incoming solar irradiance (i.e., f c_mod = ECF × cos(SZA)/SED 2 ) vs. CERES TOA shortwave (SW) flux over ocean for January 2007 showing a highly linear relationship.The color bar shows the sample density (normalized, in terms of fractional amount, in percent) for each bin in the collocated data set; when only a single point is in a bin, it is shown as a dark blue dot.R is the linear correlation coefficient.The solid black line is a linear fit to the data.

Figure 3 .
Figure 3.A schematic of the neural network model used for estimation of TOA SW flux with OMI UV retrieved parameters.The table in the bottom lists all the input parameters corresponding to two NN models used.

Figure 5 .
Figure 5. Two-dimensional histograms (similar to Fig. 2) of daily mean CERES-and OMI-derived (NN) TOA SW flux for January 2007: (a) the NN is trained on and applied to data from January 2007 and compared with CERES data from that month and (b) the NN is trained using data from January 2007 and applied with input data from January 2006 and compared with CERES data from January 2006.The color bar shows sample density (in fractional (percent) amount) of the collocated data set (see text for more details).The three dotted lines are 5, 10, and 15 % error envelope lines.

Figure 6 .
Figure 6.Two-dimensional histogram similar to Fig. 5 but showing training (top) and validation (bottom) results from the combined all-sky NN models (input parameters listed in Fig. 3, model "h" for f c < 1 and as in Fig. 3 for f c > 1) as final selected models for estimation of TOA SW flux.The data from two NN models have been combined in these plots.These are instantaneous flux values averaged over 1 • × 1 • grid boxes.Data from the 1st day of each month of 2007 are used for training and data from the 16th day of each month of 2007 are used for validation.The three dotted lines are 5, 10, and 15 % error envelope lines.

Figure 7 .
Figure 7. Root mean square errors (RMSE), normalized RMSE (NRMSE in percent), data samples (percent), and bias (percent) in training and validation data sets (same model and data as used in Fig. 6) as a function of effective cloud fraction for the data presented in Fig. 4. Model "h" is used to produce these results.

Figure 11 .
Figure 11.Similar to Fig. 10c and d but with a NN trained using data from the OMI cloud O 2 -O 2 product for July 2007.

Figure 12 .
Figure 12.(a) The daily global mean time series of TOA SW fluxes from the OMI NN and CERES and OMI-CERES (percent) on the secondary y axis.(b) Mean TOA SWF from CERES and OMI averaged along each latitude belt and OMI-CERES (percent) on the secondary x axis; (c) same as (b) but along longitude belts.The data used in (b) and (c) are from July 2007.

Figure 13 .
Figure 13.Similar to Fig. 10a, c, and d except over land for July 2007.

Table 1 .
Statistical analysis of the input parameter selection exercise.The correlation coefficient (R), slope, bias, and standard deviation (SD) of OMI-CERES TOA SW flux for eight different NN models are presented.These numbers correspond to daily intercomparison between OMI and CERES TOA SW flux.Data from January 2007 are used for this exercise.

Table 2 .
Statistical parameters corresponding daily and monthly intercomparisons of pixel and gridded TOA SW flux data from CERES and OMI.