Application of oxygen A-band equivalent width to disambiguate downwelling radiances for cloud optical depth measurement

This paper presents the three-waveband spectrally agile technique (TWST) for measuring cloud optical depth (COD). TWST is a portable field-proven sensor and retrieval method offering a unique combination of fast (1 Hz) cloudresolving (0.5 field of view) real-time-reported COD measurements. It entails ground-based measurement of visible and near-infrared (VNIR) zenith spectral radiances much like the Aerosol Robotic Network (AERONET) cloud-mode sensors. What is novel in our approach is that we employ absorption in the oxygen A-band as a means of resolving the COD ambiguity inherent in using up-looking spectral radiances. We describe the TWST sensor and algorithm, and assess their merits by comparison to AERONET cloud-mode measurements collected during the US Department of Energy’s Atmospheric Radiation Measurements (ARM) TwoColumn Aerosol Project (TCAP). Spectral radiance agreement was better than 1 %, while a linear fit of COD yielded a slope of 0.905 (TWST reporting higher COD) and offset of −2.1.


Design and characteristics 2
The heart of the TWST sensor is a zenith-pointing calibrated spectroradiometer. We elected 3 to design the sensor around a commercial compact grating spectrometer (CGS), given the 4 significant advances in miniaturization, rugged monolithic construction, and linear array 5 detectors. Several advantages accrue from this design choice, the most important to our COD 6 measurement application being the acquisition of spectral radiances at high signal-to-noise-7 ratio (SNR) and high temporal resolution, attributed to the multiplex advantage provided by 8 the CGS. 9 The key specifications for the TWST COD sensor are listed in Table 1. Here we are excluding 10 extreme ambient conditions outside the range of -10 C to +40 C that require special 11 temperature control. The spectral resolution defined by the spectrometer configuration is 12 currently ~2.5 nm (including convolution with slit function) and the sampling interval is 13 ~0.3nm. With integral order-sorting filter, the stray light level is cited to be <0.1%. The 14 temporal resolution, determined by the available sunlight, spectrometer throughput, and linear 15 FPA detector characteristics, is a variable sampling interval from 1 msec to 60 sec (1 sec 16 typical). A typical TWST spectrum recorded at 1 sec interval consists of 400 co-added 17 snapshots each of 2.5 msec integration time. The SNR for a single snapshot is limited to 400:1 18 due to photo-electron noise based on the electron well depth of 160,000. When readout noise 19 is included, this drops to 275:1. With 400 co-adds, the 1 second maximum-signal SNR is 20 therefore about 5,500:1. 21 We have now built and tested a few different configurations of the TWST sensor, but each 22 includes the basic elements represented in the schematic design in Figure 1; a companion 23 photograph looking inside a recent model is shown in Figure 2. The entrance window (A) is 24 slanted to shed rain drops. A simple mechanical shutter (S) for recording dark spectra, 25 selected for its reliability and effective light blocking performance, is located just inside the 26 input window and driven by an inexpensive stepping motor. An incoming light baffle (B) 27 limits the total field-of-view to 0.5 ° FWHM. A collecting lens (C) then focuses the filtered 28 light onto the end of a 400 µm dia. optical fiber (D) which feeds the light into the CGS (E). 29 The entire system is contained in an IP66 (NEMA 4X) rated sealed enclosure with desiccant 30 to prevent water condensation over deployment periods of several months. Our design has proven to be field-worthy, easily transportable, and stable over a wide range of environmental 1 conditions as supported in section 4.2.1. We have experienced no instances of condensation 2 inside the sealed TWST enclosure while operating in humidity and temperature conditions 3 well below the dew point. 4

Example spectral data 5
The TWST retrieval algorithm uses three spectral factors (Figure 3): the spectral radiances at 6 440 nm and 870 nm (SR440 and SR870) and the equivalent width (EQW) of the oxygen A-7 band (section 3.3.1). Figure 3 shows example calibrated spectral measurements for nearly 8 identical SZA, but for clear sky and a range of COD values in the Thin optical thickness 9 regime. The overall radiance level as well as the depth of the oxygen A-band absorption are 10 observed to increase with COD. 11

Calibration and dark current correction 12
There are two forms of calibration that must be managed for any technique that uses spectro-13 radiometers: wavelength and radiometric. The wavelength calibration of the compact grating 14 spectrometers used in our TWST sensors has proven stable over periods of months. 15 Furthermore, the TWST approach does not rely on resolving spectral line structure. The 440 16 nm and 870 nm spectral radiance levels, due to their shallow spectral slopes (Figure 3), and 17 A-Band EQW value, due to its accumulation over many spectral bins, are relatively 18 insensitive to foreseeable thermal shifting of the spectral sampling grid. 19 TWST spectral radiance calibration is performed at the beginning and end of every field 20 deployment and more frequently as needed. Our calibration source is a Labsphere Uniform 21 Radiance Standard integrating sphere. It is well-known that the standard incandescent source 22 lamps age and need to be replaced periodically. Like other long time users, we find these 23 lamps to be the largest source of uncertainty and absolute error in our radiometric calibration 24 procedure; that uncertainty is ~5%. During each calibration we set the integration period, 25 number of snapshot coadds, and aperture radiance to span the range of field conditions 26 anticipated for sunlit clouds, and then we derive a linear photoresponsivity coefficient in the 27 usual manner. These radiometric calibration records for each TWST unit are kept and 28 compared over periods of years to monitor the stability of each unit for its lifetime. Having 29 records for some units over 2-4 years, we find changes in the calibration of 1-3%, well within 1 the uncertainty of our calibration lamps which as noted above is on the order of 5%. 2 The spectrometer's silicon CCD detector outputs are susceptible to offset drift, typically 3 driven by changes in ambient temperature, but the detector array contains light-shielded dark-4 reference detectors intended to automatically track and subtract such drift. In addition, TWST 5 employs a mechanical shutter for frequent collection of dark spectra, typically a 1-second 6 dark spectrum every 60 seconds. The dark correction (offset subtraction) applied to each 7 recorded spectrum is spline-extrapolated from earlier collected dark spectra. In section 4.2.1, 8 the effectiveness of these calibration methods is evaluated by comparison to coincident 9 AERONET spectral radiance observations over a period of several weeks. 10 3 TWST cloud optical depth retrieval method 11

Section outline 12
The TWST retrieval algorithm employs model-generated look-up tables to convert zenith 13 spectral radiance at 440 nm (SR440) to a numerical value of COD. However, the TWST 14 algorithm first determines the cloud optical thickness regime, Thin or Thick, and thus whether 15 to reference the thin or thick branch of the SR440 to COD look-up table. We first discuss the 16 somewhat conventional spectral radiance to numerical COD lookup, including table  17 generation, COD error sensitivity to principal uncertainties via radiative transfer simulations, 18 and technique of interpolation between table entries. Then we discuss the determination of 19 the optical thickness regime. This entails discussion of why the Oxygen A-band and its 20 equivalent width (EQW) metric are informative of the thickness regime. We introduce the 21 "nose" plot of SR440 vs EQW, and its generic slope characteristics are revealed as a key to 22 resolving the thin/thick ambiguity; the algorithm does not use model-generated nose plots. 23 We explain the need and basis for the SR440/SR870 ratio as color index. The color index and 24 nose plot slope metrics are combined in a multivariate temporal filter that continually updates 25 the estimate of optical thickness regime. To illustrate its operation, and how it copes with 3D 26 cloud effects, we discuss an example nose plot time sequence and filtered results.
3.2 Numerical spectral radiance to COD lookup 1 3.2.1 Radiative transfer construct 2 Assuming the sensor's field of view (FOV) does not include the sun, the zenith spectral 3 radiance consists of solar radiation scattered by the molecules, aerosols, and cloud water 4 droplets in the FOV, which may include radiation that has been scattered multiple times from 5 the atmosphere and the terrain. The visible/NIR spectral band (Figure 3), at a moderate 6 spectral resolution of 2 nm, shows a broad baseline with multiple narrow absorption features. 7 Many of these are due to water vapor, as well as Fraunhofer lines. The spectral radiance at 8 440 nm is in a region relatively free from atmospheric gaseous absorption, and thus suitable 9 as a COD proxy. We chose 440 nm for TWST radiance-to-COD look-up because that is a 10 wavelength used by AERONET Cloud-Mode sensors, which serve as a source for 11 comparative validation. 12 The model used for generating 440 nm radiance-to-COD look-up tables is the MODTRAN5 13 atmospheric radiative transfer code (Berk et al., 2006). MODTRAN incorporates the DISORT 14 code (Stamnes et al, 1988) for plane-parallel stratified media, i.e., idealized 1-dimensional 15 radiative transfer (1DRT). Calculations are done for a typical water stratus cloud above a 16 stated ground albedo, for a stated nominal aerosol profile, over a grid of COD and solar zenith 17 angles (SZA). Figure 4 is a graphical depiction of sample tables. For any SZA, there is a 18 "bright point" radiance where the idealized 1DRT cloud radiance reaches a maximum, 19 typically occurring for a COD between 2 and 8, as seen in Figure 4. Real clouds manifest 3-20 dimensional radiative transfer (3DRT) effects, including radiances exceeding the idealized 21 1DRT bright point radiances (Marshak et al, (2000)). When faced with such exceedances, the 22 TWST retrieval algorithm reports the COD corresponding to the bright point radiance but 23 flags an out-of-bounds ("3D Cloud") condition. 24 3.2.2 COD error sensitivity to radiative transfer parameter uncertainties

25
The TWST algorithm currently operates without any information on the droplet size 26 distribution or the cloud base height, and with a prior estimate of the ground albedo and 27 aerosol loading profile. We do not consider deviation of the actual from nominal aerosol 28 profile, as such perturbation from the baseline aerosol optical depth (AOD) is typically a 29 small contributor to reported COD. We performed some initial sensitivity studies on these 30 remaining parameters. The albedo sensitivity findings below will prove of value in helping to explain the minor disagreement bias between coincident TWST and AERONET COD 1 observations (section 4.2.2). 2 Because our implementation of the TWST algorithm uses a radiance database generated with 3 the MODTRAN model, we studied 440 nm radiances from four different cloud types 4 parameterized within MODTRAN, which assume Mie scattering, log-normal droplet size 5 distribution, and liquid water refractive index. These types have effective radii of 12.0 6 (cumulus), 7.2 (altostratus), 8.3 (stratus), and 6.7 (stratocumulus) µm. Water cloud drop-size 7 distributions typically vary from an effective radius of 1 -20 µm (see e.g. Chiu et al, (2006)). 8 We modeled clouds with fixed base height of 0.5 km and fixed physical thickness of 0.5 km. 9 For each cloud type we varied the Liquid Water Path (LWP) enough to achieve 550 nm CODs 10 between 0 and 100; LWP was used because it is an input to MODTRAN. COD values were 11 estimated from LWP using the Wood and Hartmann, (2006)  thin/thick ambiguity, the strong variation with solar zenith, and the weaker variation with 30 ground albedo are evident in these plotted results. 31 The plot in Figure 8(a) further explores this sensitivity, and shows the signed change in 1 retrieved COD value for an unexpected increase in the ground albedo from 0.1 (for which the 2 radiance-to-COD lookup tables are computed) to 0.2. Each curve, for either thick or thin 3 cloud, pertains to some fixed percentage of the aforementioned 1DRT bright point radiance 4 "Lbrt" (which varies with SZA, cf. Figure 4). These curves show that a higher than expected 5 albedo implies retrieval of a lower (higher) COD in the Thick (Thin) regime. The largest 6 change we found was ΔCOD of 5, but that occurred for very thick clouds, such that the 7 relative change was only 10%. Near the bright point (red curves in Figure 8(a)) the COD vs 8 radiance is quite non-linear (Figure 4), and thus the red curves, approximated by linear 9 interpolations for this study, are less-precise, and jagged. 10 To provide analytic support to these albedo sensitivity findings, we performed calculations 11 employing asymptotic radiative transfer (ART) theory relations as elucidated by King, (1987 shown in Figure 9. The reader is directed to graphically determine the error in retrieved COD 20 value by first starting with a given true COD value, tracing rightward from that y-intercept 21 parallel to the x-axis and reaching a curve pair for a given SZA. The right curve of the pair 22 registers the actual radiance measured for the unexpected 0.2 albedo, but the left curve is 23 radiance-to-COD lookup table computed for the expected 0.1 albedo. So at intersection with 24 right curve, the reader traces down parallel to the y-axis to intercept the left curve, then traces 25 leftward back to the y-axis and reads out the lower COD value. The solid curves of Figure  26 8(b) are not exactly comparable to those of Figure 8(a), in that the former are for constant-27 COD whereas the latter are for constant relative radiance (with respect to the 1DRT bright 28 point radiance). To corroborate that the COD error in fact decreases with SZA, the dashed 29 curves of Figure 8(b) better (but not exactly) correspond to constant relative radiance. These 30 dashed curves have COD decrease with decreasing SZA, which referring to Figure 4 yields a At this point it is important to note that the spectral radiance chosen at some other wavelength 1 than 440 nm could be used for the radiance-to-COD look-up. In principle the choice of 2 wavelength depends upon freedom from atmospheric gaseous absorption and on the ground 3 albedo of the measurement site. One wants a wavelength with the lowest absolute albedo 4 uncertainty to minimize errors in the COD due to errors in the assumed albedo for the 5 MODTRAN5 computations. This flexibility in the choice of wavelength is basis of the term 6 "Spectrally-agile" within the TWST acronym. 7 For low-altitude water clouds, uncertainty in the cloud base height has a negligible impact on 8 COD retrieval. We ran MODTRAN for cloud base heights of 500 m and 2 km, iterating over 9 10 COD and 11 solar zenith angles for each. The 440 nm radiances were nearly identical, as 10 shown in the scatter plot of Figure 10.  (Figure 11). Thus its continua-1 normalized (section 3.3.2) spectral-average quantity, termed the Equivalent Width (EQW), 2 provides a direct measurement of the average amount of oxygen-density-weighted photon 3 path length from the sun to the sensor. Since oxygen is uniformly mixed in the atmosphere, 4 this is related to the photons' physical path lengths. Therefore the EQW supplies useful 5 information about whether a zenith radiance measurement is in the optically-thin or optically-6 thick regime. A virtue of EQW is that it may be stably computed from low-resolution spectral 7 data such as from the TWST sensor, as detailed in section 3.3.2. 8 Of course, other factors cause EQW to change besides COD. Changes in the SZA produce 9 decreases in EQW with time during the morning and increases in the afternoon. Changes in 10 the density-weighted average cloud thickness and cloud altitude also affect the EQW 11 independent of the COD. is a monotonic function of COD for COD~>1. By plotting SR440 versus EQW as COD 24 increases from no cloud to thick clouds one traces out a "nose-like" shape ( Figure 12). For the 25 very lowest COD values, EQW decreases with increasing COD and the slope is negative. 26 Beyond about COD=1, the lower portion of the "nose" where the slope is positive corresponds 27 to the optically thin regime; the upper portion where the slope is negative corresponds to the 28 optically thick regime. Within a span of several seconds, passing clouds most often do not 29 trace out a complete nose but only a small segment of it as the cloud evolves and drifts in the 30 wind over the sensor. Notwithstanding the lowest COD values discussed further below, whether the cloud changes involve increases or decreases of the COD, the slope of the 1 corresponding segment indicates the cloud's optical thickness regime. This is the TWST basis 2 for resolving the COD ambiguity. 3 The nose plot in Figure 12 includes a smooth curve, based on MODTRAN5 computations but 4 elastically stretched to fit the depicted data points over a 4 minute measurement where the 5 COD varied strongly between the indicated blue sky, thin and thick regimes, as well as points 6 deviating well away from the ideal 1DRT smooth curve. These deviating points are classified 7 as either "3D Cloud" based on their SR440 exceedance of the 1DRT bright point radiance 8 value (section 3.2.1) or as "Mixed" points attributed to heterogeneous cloud structure within 9 the field of view, itself a 3DRT effect. The classification of the remaining data points into 10 optically thin, thick or blue sky regimes was corroborated against coincident all sky camera 11 video. Although the MODTRAN5 computed nose plot curve supports these regime 12 classifications, it is important to note that the TWST algorithm does not employ model-13 generated nose plot curves to guide its thickness regime determination. Indeed, the particular 14 shape and slope of a computed nose plot curve varies, as it should, with the unknown physical 15 cloud thickness. Instead, the algorithm exploits the aforementioned positive-slope:Thin, 16 negative-slope:Thick generic properties of the nose plot. 17

18
The cloud optical thickness regime determination operates in two distinct radiance domains. 19 When the COD is very low, e.g. COD<=1, the amount of radiation in the NIR is very small 20 and the signal to noise ratio (SNR) of the EQW is low (e.g. "clear sky" in Figure 3) and thus 21 the nose plot slope SNR is too low to resolve the thickness regime. The ratio of SR440 to 22 SR870, termed the color index, has a much higher SNR and is more reliable in this regime. 23 This, of course, is a simple consequence of the wavelength dependence of Rayleigh 24 scattering. For low-moderate altitude water clouds, small-moderate SZA, and typical 440 nm 25 ground albedos less than 0.2 ( Figure 6), our data analyses have found a hard threshold of 4 to 26 be a sure indicator of optically very thin clouds (e.g. "clear sky" in Figure 3) and a soft 27 threshold of 2<index<4 to be a strong indicator (e.g. "COD<1" in Figure 3). When the color 28 index is less than 2, the cloud's optical thickness is not well correlated with the index, and the 29 algorithm must rely on the nose plot slope. Figure 13 re-depicts the nose plot data points of Figure 12, this time connecting a subset of 1 points with line segments to indicate adjacent samples in a 2-minute time series. If measured 2 nose plots followed an idealized 1DRT curve as indicated in Figure 12, the determination of 3 thickness regime would be nearly trivial. A linear regression over a short time segment would 4 suffice. Clearly more complex logic is required, yet it is visually evident that local coherence 5 could be exploited. The qualitative reduction in ambiguity afforded by examining a sufficient 6 time record suggests the use of a filter with memory. For example, for passing or evolving 7 clouds spatially well-resolved within a narrow field of view, the thickness regime should not 8 switch rapidly between thick and thin except possibly near the bright point (thick/thin regime 9 boundary) where a switch is inconsequential to the retrieved COD. We implemented a time 10 varying hysteresis filter to effectively avoid this unwanted switching. The hysteresis action is 11 achieved by keeping track of the maximum and minimum values of equivalent width over a 12 predetermined time interval, typically about three minutes. In order to drive the filter toward 13 a different thickness regime, the hysteresis limits must be exceeded. The output of the 14 hysteresis filter is discrete ternary: -1, 0 or 1, corresponding to thick, indeterminate or thin. 15 Finally, this ternary variable is input to a linear, single pole autoregressive (AR(1)) filter to 16 afford additional smoothing. The output of this filter is thresholded and used as the thickness 17 regime estimate. a radiance of about 23). The EQW values of those points are reasonable, but the radiances are 31 lower than expected. Our supposition at this time is that some of these are due to further 3D 32 cloud structure effects, but leading to darker radiances rather than the bright radiances of the 1 yellow points, while others points may be due to spatial averaging over the field-of-view. 2  with a 4 second difference between the high gain (x8) A and low gain (x1) K measurements 28 from AERONET. The result ( Figure 14) shows that both sensors were reporting SR440 values 29 in very good agreement. The rms difference was 0.63 (µW cm -2 sr -1 nm -1 ). A simple linear fit 1 without constant yielded a slope of 1.003 (0.0004). TWST values at high spectral radiance 2 showed some evidence of nonlinear response. 3 Several conclusions follow from the very good agreement among TWST and AERONET 4 spectral radiances. The first is the expectation of a COD comparison not influenced by TWST 5 spectral radiance errors. As a corollary, the COD comparison should not be unduly influenced 6 by different fields of view (1.2 ° for AERONET versus 0.5 ° for TWST) and zenith pointing 7 (robotic control for AERONET versus fixed tripod with bubble level for TWST), given the 8 close agreement over many different cloud conditions. The sensors were laterally displaced 9 by about 3 m, and for a 1 km cloud base altitude their field of view footprints are 20 m and 8 10 m. Of course, the agreement only proves consistency, not accuracy for either sensor. The 11 second is the radiometric stability of TWST during its TCAP deployment. This is 12 corroborated by the stability of the 4 pre-and 1 post-test radiometric calibrations, with the 13 photoresponsivity coefficient at 440 nm for 9 July being 98.1% of that for 17 May. Of the 244 overlapping COD values, 235 (96%) showed the same cloud thickness regime. 28 Some analysis was done in an effort to determine whether TWST or AERONET Cloud-Mode 29 was probably correct. For the nine cases where AERONET and TWST disagreed on the 30 thickness regime, detailed nose plots were generated to see if we could visually extract more than the simple slope information used in the automated algorithm. Four of the cases 1 produced close to the ideal nose shape, indicating that the TWST thickness regime was 2 probably correct. For the other five cases, the nose plot was too distorted to determine the 3 thickness regime, indicating that the TWST thickness regime was probably incorrect and 4 should have been assigned the "unknown" label. 5

Measurements and comparisons to AERONET
A linear fit of TWST to AERONET Cloud-Mode COD, for the 235 cases of thickness regime 6 agreement, for fixed zero-intercept, found a slope of 0.843 (TWST reporting higher COD 7 values) with an rms difference of COD 3.2. This was repeated while dropping the two high 8 COD value outlier points (Figure 16), but the slope only changed by 1%. No evidence of a 9 constant offset between TWST and AERONET Cloud-Mode was found. However, the 10 sparsity of such evidence is due to the relatively few optically thin COD cases available from 11 AERONET, due to the secondary mission status of its Cloud-Mode. (When skies are largely 12 clear, AERONET executes its primary mission of aerosol optical depth and microphysical 13 property retrieval measurements). Therefore, another linear fit, this time with free-intercept, 14 found a slope of 0.905 and constant offset of -2.1. 15 The two primary candidates for causing the observed disagreements are differences in the 16 TWST and AERONET Cloud-Mode lookup tables and effects from the trimmed mean 17 process. There may also be some residual effects due to FOV and pointing differences, 18 although these are not expected to be large due to the very good spectral radiance agreement 19 (section 4.2.1). A partial explanation centering on the lookup tables is the difference in 20 assumed ground albedos between the sensors. The TWST SR440-to-COD lookup table 21 generated from MODTRAN used a weighted average of water, deciduous vegetation, dead 22 pine, and sand albedos, resulting in an earth albedo at 440 nm of 0.078545. On the other 23 hand, AERONET updates its ground albedo episodically every few days from MODIS data 24 products or a (rolling) 16-day average MODIS historical database (Chiu et al, (2012)). For 25 this dataset, the AERONET-employed albedos were lower than that assumed for TWST, 26 varying between 0.02-0.04, 0.03 average. Most of the sample points are in the optically Thick 27 regime, and according to our albedo sensitivity discussion (section 3.2.2), a lower than 28 expected albedo implies TWST retrieval of higher COD values in the Thick regime, 29 consistent with the linear fits. Figure 8 depicts an approximately constant relative COD assumed albedo therefore explains about half (0.05) of the difference between a slope of unity 1 and the fitted slope (0.905). 2 It must always be kept in mind that the COD values determined by TWST and AERONET 3 Cloud-Mode are only equivalent 1DRT quantities. Violations of 1D assumptions are present 4 in nearly all our measurements to some extent. This includes (1) cases where the observed 5 spectral radiance is greater than that possible for any 1D cloud, (2) cases where deviations 6 from the ideal nose plot are too high for any 1D cloud and (3) the many cases where the rapid 7 variation in spectral radiance is too high for 1D clouds. 8

Conclusions 9
Overall the good agreement between TWST and AERONET Cloud-Mode Cloud Optical 10 Depth values, across many weeks of coincident field deployment, validates the performance 11 of the Three Waveband Spectrally-agile Technique as well as the field-worthiness of the 12 TWST sensor. Although the spectrally-agile nature of TWST was not investigated in this 13 study, its advantage over fixed spectral bands for cases with high albedo at 440 nm may be 14 the focus of a future study. Although our error sensitivity studies in section 3.2.2 and the 15 agreement with AERONET over many weeks of the TCAP campaign lend confidence in 16 applying TWST for nominal conditions, future efforts will ascertain and, where possible, 17 extend the operational limits (e.g., SZA, ground albedo) of the TWST retrieval algorithm. 18 One of the most notable results of our experience with TWST is the high signal-to-noise ratio 19 available in the high temporal (1 second), spatial (0.5 ° field of view), and spectral (2.5 nm) 20 resolution data TWST generates. At peak signal, at a COD value of approximately 5, the SNR 21 is estimated to be 5,000:1 for 1 Hz reports.