High-Dynamic-Range Imaging for Cloud Segmentation

Sky/cloud images obtained from ground-based sky-cameras are usually captured using a fish-eye lens with a wide field of view. However, the sky exhibits a large dynamic range in terms of luminance, more than a conventional camera can capture. It is thus difficult to capture the details of an entire scene with a regular camera in a single shot. In most cases, the circumsolar region is over-exposed, and the regions near the horizon are under-exposed. This renders cloud segmentation for such images difficult. In this paper, we propose HDRCloudSeg -- an effective method for cloud segmentation using High-Dynamic-Range (HDR) imaging based on multi-exposure fusion. We describe the HDR image generation process and release a new database to the community for benchmarking. Our proposed approach is the first using HDR radiance maps for cloud segmentation and achieves very good results.

While there have been several studies analyzing clouds and their features from WSI images (Long et al., 2006;Souza-Echer et al., 2006;Li et al., 2011;Liu et al., 2015a;Dev et al., 2014a), most of them avoid the circumsolar region, because capturing the details in this area is a non-trivial task. The region around the sun has a luminous intensity several orders of magnitude higher than other parts of the scene. The ratio between the largest and the smallest luminance value of a scene is referred to as its Dynamic Range (DR). On a typical sunny and clear day, it is difficult to capture the entire luminance range of the sky scene using a low-dynamic-range (LDR) image. Yankee Environmental Systems sells a well-known commercial sky camera model (TSI-880) (Long et al., 2001). It is a fully automated sky imager, used in continuous monitoring of the clouds. Its onboard processor computes the cloud coverage and sunshine duration, and stores these results for the user for further processing.
One of the earliest attempts to capture more of the dynamic range of the sky was done by Stumpfel et al. (2004). They presented a framework in which a set of exposure settings along with neutral density filters are used to generate an HDR composite map. Kenny et al. (2006) used a digital camera to estimate the whole-sky luminance distribution for different sky conditions. Attempts to provide a full spherical HDR view of the sky/cloud condition were done by mounting hemispherical sky cameras on the top and bottom of airships (Okura et al., 2012). Gryaditskya et al. (2014) used HDR captures of the sky to recover absolute luminance values from images. To the best of our knowledge, there is no prior work that uses the capability of HDR imaging for better segmentation of sky/cloud images.
Several techniques for cloud segmentation exist, but they are designed for conventional LDR images. Long et al. (2006) developed a method based on fixed thresholding. As clouds have a non-rigid structure, traditional segmentation algorithms based on shape priors are not applicable. Color is generally used as a discriminating feature in cloud segmentation (Li et al., 2011;Souza-Echer et al., 2006;Mantelli-Neto et al., 2010). Li et al. (2011) use a hybrid thresholding approach that employs both fixed and adaptive thresholds, depending on the bimodality of the input image. Long et al. (2006) model atmospheric scattering by calculating the ratio of red and blue color channels. Souza-Echer et al. (2006) define appropriate thresholds in the saturation channel of the intensity-hue-saturation (IHS) color model. Mantelli-Neto et al. (2010) uses a multi-dimensional Euclidean distance to determine the locus of sky and cloud pixels.
The motivation of this paper is to propose high-dynamic-range imaging for cloud segmentation using ground-based sky cameras. We detail the process of image capture and storage in our sky camera system. We show that using HDR cloud imaging significantly reduces the amount of saturated pixels in an image, and is therefore an efficient manner to capture the circumsolar region. Furthermore, HDR imaging generally provides better segmentation results, as compared to LDR images, regardless of the segmentation method used. In this paper, we show how to improve segmentation results by capturing a larger dynamic range of the sky using High-Dynamic-Range Imaging (HDRI) techniques. We then introduce HDRCloudSeg, a graph-cut based segmentation algorithm that uses the HDR radiance map for accurate segmentation of sky/cloud images. The main novel contributions of the present manuscript include: -Introducing an HDR capture process for better cloud imaging as compared to conventional LDR images. This includes methods and procedures to capture a larger part of the dynamic range of a sky scene and archiving the obtained images efficiently; -Proposing a graph-cut based segmentation algorithm that offers better performance as compared to the state-of-the-art approaches; -Releasing a database comprising high dynamic captures of the sky scene, along with their manually annotated groundtruth segmentation maps.
The rest of this paper is structured as follows. We discuss the process of HDRI generation in Section 2. We introduce HDRCloudSeg in Section 3 and evaluate it in Section 4. Section 5 concludes the paper.

HDR Image Generation
WAHRSIS, our custom-designed sky imagers, are composed of a Digital Single-Lens Reflex (DSLR) camera with a fish-eye lens. The camera is controlled by a single-board computer housed inside a box with a transparent dome. Our first model (Dev et al., 2014b) used a moving sphere mounted on two motorized arms to block the direct light of the sun. We removed this from our latest models in favor of HDR imaging (Dev et al., 2015b). In this way, we avoid potential mechanical problems as well as occlusions in the resulting images.
Typical cameras can only capture an 8-bit low dynamic range (LDR) image. Therefore, we use the exposure bracketing option of our cameras to capture three pictures at 0, −2 and −4 exposure value (EV) in quick succession. The aperture remains fixed across all captured images, while the shutter speed automatically adapts to match the appropriate exposure. Figure 2 shows an example of the captured images at varying exposure levels. An LDR image taken at low exposure (cf. Fig.2(c)) has few saturated pixels in the circumsolar region, but is underexposed in the regions further from the sun. On the other hand, a high-or medium-exposure LDR image (cf. Fig.2(a) and (b)) can capture the near-horizon regions well, but the circumsolar area is overexposed and saturated. The different LDR images can then be fused together into a single, high-dynamic-range (HDR) radiance map. We use Debevec and Malik (1997)'s algorithm to recover the HDR radiance map from LDR images. While the radiance map can be analyzed computationally, it cannot be viewed directly on a conventional (LDR) display. Therefore, for the purpose of visualization, we tone-map the HDR radiance map into a conventional 8-bit LDR image using contrast-limited adaptive histogram equalization (CLAHE) (Zuiderveld, 1994). It involves a logarithmic transformation of the higher DR radiance map to a lower 8-bit display. This produces a perceptually recognizable image, because the response of the human eye to light is also logarithmic. This operation compresses the dynamic range by preserving the details in the image. Figure 2(d) shows an example of a tone-mapped image.
We also perform the HDR fusion and tone-mapping on a server that collects the images from all our WSIs. Since we have three imagers capturing multiple exposures every 2 minutes, we need algorithms which are computationally efficient.
Therefore, we use the GPU implementation as described by Akyüz (2015). It relies on the OpenGL API to perform the entire HDR pipeline on the GPU. It consists of HDR fusion, which is detailed in Akyüz (2015), followed by tone-mapping, which is based on the photographic tone reproduction operator of Reinhard et al. (2002). Generating an 18-Megapixel HDR image and its tone-mapped version takes less than 7 seconds, compared to several minutes with standard CPU algorithms.
For storage efficiency, we compress our HDR images using the JPEG-XT format (Richter, 2013). 1 JPEG-XT is an extension of JPEG currently being developed for HDR images. An important advantage of JPEG-XT format is that it is backward compatible to the popular JPEG compression technique. It consists of a base layer that is compatible with the legacy systems and an extension layer that provides the full dynamic range. Using JPEG-XT at a quality level of 90%, we obtain a file size of about 8 MB. This represents a significant improvement compared to the common RGBE format, which would require 50MB per image by storing every pixel with one byte per color channel and one byte as a shared exponent. 1 We use the reference software source code available at https://jpeg.org/jpegxt/software.html We propose a graph-based segmentation algorithm called HDRCloudSeg that formulates the sky/cloud segmentation task as a graph-partitioning problem. Graph-cut for cloud segmentation was proposed earlier by Liu et al. (2015b). However, Liu et al. used conventional LDR images in their segmentation framework and generated seeds using Otsu threshold (Otsu, 1979).
Furthermore, they did not consider the circumsolar regions in their evaluation.

Notation
Suppose that the low-, medium-and high-exposure LDR sky/cloud images are represented by X L i , X M i and X H i respectively, i = 1, . . . , N , where N is the total number of HDR sets in the dataset. Without any loss of generality, X i denotes a low-, medium-, or high-exposure LDR image in the subsequent sections. Each of these LDR images are RGB color images, X i ∈ IR a×b×3 , with dimension a × b for each R, G and B channel. We generate the HDR radiance map H i ∈ IR a×b×3 from the set of three LDR images X L i , X M i and X H i as described in Section 2. Let p mn be a sample pixel in the image X i , where m = 1, . . . , a and n = 1, . . . , b. We aim to assign labels to each of the pixels p mn , as either cloud or sky. We denote the label as L p , where L p = 1 or 0 if p mn is a cloud or sky pixel, respectively.
We model this task as a graph-based discrete labeling problem, wherein we represent X i as a graph, comprising a set of nodes and edges.

Generating Seeds
Graph-cut based segmentation algorithms Kolmogorov and Zabin, 2004;Boykov and Kolmogorov, 2004;Bagon, 2006) require the user to initially label a few pixels as 'foreground' and 'background'. We refer to these prior labeled pixels as seeds. The process of generating seeds is generally done manually before partitioning the graph into two sub-graphs.
In HDRCloudSeg, we automatically generate these initial seeds by assigning a few pixels in the sky/cloud image with sky and cloud labels.
Most sky/cloud segmentation algorithms (Li et al., 2011;Long et al., 2006;Souza-Echer et al., 2006;Mantelli-Neto et al., 2010) use color as the discriminating feature to distinguish cloud from sky. In our previous work, we have performed a systematic analysis of existing color spaces and components for conventional LDR images and observed that color channels such as (B − R)/(B + R) (B and R indicating the blue and red channels of the image) are among the most discriminative for cloud segmentation. We discuss further details on discriminatory color channels of HDR luminance maps in Section 4.2.1. Furthermore, we have proposed a probabilistic approach for sky/cloud image segmentation, instead of the conventional binary approach. In this probabilistic approach, each pixel is assigned a soft membership to belong to the cloud category, instead of a hard membership (Dev et al., 2014a).
To illustrate these concepts, consider the example shown in Fig. 3. Figure 3(a) shows a sample LDR sky/cloud image X i .
We extract the (B −R)/(B +R) color channel from this LDR image as shown in Fig. 3(b). A fuzzy C-means clustering on this extracted color channel yields the probabilistic output image, shown in Fig. 3(c). It denotes the confidence level of each pixel to belong to the cloud category. We assume that for a given pixel, the sum of membership values for sky and cloud category is unity. In HDRCloudSeg, we apply fuzzy C-means clustering to the most discriminatory color channel of the HDR radiance map H i to estimate the probability of a pixel belonging to the cloud category. We denote the ratio channel as Y i . The membership values obtained by clustering provide us a mechanism for assigning the seeds with a degree of confidence. We assign these initial seeds for HDRCloudSeg as follows: pixels having membership >α or <(1 − α) are labeled as cloud and sky respectively.
The value of α is a constant and is set experimentally (more on this below).

Partitioning the HDR Graph
HDRCloudSeg employs a graph-based image segmentation approach, wherein we represent the ratio-image Y i ∈ IR a×b as a set of nodes and edges. Each edge of the graph is given a corresponding weight that measures the dissimilarity between two pixels. Such methods are based on pixel adjacency graphs, where each vertex is a pixel and the edges between them are defined by adjacency relations. We follow the work of Boykov and Jolly (2001) and try to minimize the segmentation score E: where R p (A p ) denotes the data cost for an individual pixel p, B p,q denotes the interaction cost between two neighboring pixels p and q in a small neighborhood N , and µ is a weight.
The complete proposed HDR segmentation algorithm is summarized in Algorithm 1.
We illustrate the complete HDRCloudSeg segmentation framework in Fig. 4. Figure 4(a) represents the three sample LDR images X L i , X M i and X H i respectively captured at varying EV settings. We generate the corresponding HDR radiance map H i from these LDR images. A tone-mapped version of H i is shown in Fig. 4(b), for visualization purposes. We extract the (B−R)/(B+R) ratio channel from H i , and generate the initial cloud and sky seeds marked in green and red color respectively, as shown in Fig. 4(c). The binary output image after graph cut is shown in Fig. 4(d).

Algorithm 1 HDRCloudSeg
Require: LDR sky/cloud images with varying EVs. 1: Create HDR radiance map Hi from bracketed set of LDR images; 2: Extract (B − R)/(B + R) ratio channel Yi from HDR radiance map Hi; 3: Perform fuzzy C-means clustering on the extracted ratio channel Yi from HDR radiance map Hi, to generate the probabilistic map; 4: Generate initial seeds from the computed probabilistic map for image segmentation as described in Section 3.2; 5: Partition the ratio channel Yi into two subgraphs using the generated seeds; 6: return Binary segmented image.

HDR Sky/Cloud Image Database
Currently there are no available HDR image datasets for sky/cloud images. Therefore, we propose the first HDR sky/cloud dataset to the research community. We refer to this dataset as Singapore HDR Whole Sky Imaging SEGmentation dataset (SHWIMSEG). The SHWIMSEG dataset consists of 52 sets of HDR captures, comprising a total of 156 LDR images. Each HDR capture is based on three LDR images (low-, medium-and high-exposure) which were captured in Automatic Exposure Bracketing (AEB) mode of our camera. These high-quality HDR images are captured with our sky imagers, located on the rooftops of a building at Nanyang Technological University Singapore. We crop a square region of 500 × 500 pixels from each images, with the sun location as the geometrical center of the cropped image. The corresponding ground truth images for these crops were manually generated in consultation with experts from the Meteorological Service Singapore (MSS).
We release this entire HDR image dataset on sky/cloud image segmentation. 2 The SHWIMSEG dataset comprises the following: (a) original full-size LDR fish-eye images captured in three exposure settings, (b) corresponding 500 × 500 cropped images of the circumsolar region, in all the exposure settings, (c) HDR radiance maps of the crops generated via exposure fusion, (d) tonemapped images (used during visualization purposes), and (e) manually annotated binary ground-truth maps for the cropped images. We use this newly created dataset to perform a detailed evaluation of several cloud segmentation algorithms below.

Results
HDR imaging is an effective technique for cloud observation, as it helps us better image the circumsolar region with reduced overexposure. We illustrate the advantage of HDR imaging in reducing the saturation by calculating the number of saturated pixels in the low-, medium-and high-exposure LDR images. We also calculate the number of saturated pixels (if any) in the radians maps in the dataset. 3 Using our HDR techniques, we observe that the radiance maps have 24 times fewer saturated pixels, as compared to the high-exposure LDR images, and 4 times fewer saturated pixels with respect to medium-LDR images.
A reduced amount of saturated pixels is an important factor for cloud analysis, especially around the circumsolar region.
In addition to containing fewer saturated pixels, HDR imaging also helps in improved cloud segmentation performance, regardless of the techniques used, as we will demonstrate in the following. Cloud segmentation is essentially a binary classification problem, wherein we classify each pixel as either sky or cloud. We measure the classification performance of the different cloud segmentation methods using Precision, Recall, F-score and Error values, which are defined as: where TP, TN, FP, and FN denote the true positive, true negative, false positive and false negative samples in a binary image.

Color Channel Selection
Like in our previous work (Dev et al., 2014a), we study various color channels and models that are conducive for sky/cloud image segmentation. We consider 16 color models and components that are generally used in sky/cloud image segmentation. Table 1 lists the various color channels: it contains the color models RGB, HSV , Y IQ, L * a * b * , various red and blue ratio channels, and chroma (Dev et al., 2017).
We designed our proposed HDRCloudSeg segmentation algorithm to work on the HDR radiance maps. We extract the 16 color channels (as indicated in Table 1) from the HDR radiance map and perform fuzzy C-means clustering on the extracted Table 1. Color spaces and components used in our analysis. We intend to find the best color channel for HDR imaging.
color channel. We assign the initial seeds for cloud and sky pixels in the fuzzy-clustered image (more details on the seeding level in the subsequent Section 4.2.2). Figure 5 shows the segmentation error for all the 16 color channels. We observe that the saturation color channel (c 5 ) and the ratio color channel (c 15 ) has better performance, as compared to other color channels. Therefore, we choose the

Seeding Level Sensitivity
As described in Section 3.2, the value of α determines the amount of initial seeds in HDRCloudSeg. We set a high value of α, because it corresponds to higher confidence in assigning the correct labels. A higher confidence ensures low error and high accuracy. We show the Receiver Operating Characteristics (ROC) Curve of our proposed algorithm, for varying values of the seeding parameter. The ROC curve in Figure 6(a) plots the False Positive Rate (FPR) (= F P F P +T N ), against the True Positive Rate (TPR) (= T P T P +F N ). We generate these points for varying values of α in the interval [0, 1]. This curve helps the user to select the best value of the seeding parameter. In order to illustrate its effect on the segmentation performance, we also evaluate the dependency of the seeding parameter on the various evaluation metrics.  Figure 6 shows the impact of the seeding parameter α on the average performance of our segmentation framework. We report the average error percentage, precision, recall and F-score across all the images of the dataset. We observe from Fig. 6(b) that the average error gradually decreases with increasing value of the seeding parameter, α. This makes sense as higher value of α indicate higher confidence for accurate detection of labels. Similar observations apply to average precision, recall and F-score.
The sensitivity of the seeding parameter can also help us in understanding different types of clouds around the sun aureole.
In this paper, we deal only with the detection of thick clouds in the circumsolar region. Primarily this is because classifying the type of clouds into thick or thin clouds is an extremely difficult task in the area around the sun. Most of circumsolar region pixels are close to saturation. However, the seeding parameter α has a control on the type of detected clouds. For example, we can detect thin clouds by tuning the seeding parameter to favor higher cloud detection (i.e. more tendency to classify a pixel as cloud), which comes at the expense of reduced precision, however.
From Fig. 6(b), we observe that there is a dip in the Error at α = 0.88. Moreover, at this value, there is a good trade-off between precision and recall values -both are high. Therefore, we set the value of the seeding parameter α to 0.88 in the subsequent experiments. We also observe that there are a few deviation points along the curve. The minor peaks and troughs in Fig. 6 are due to the sensitivity of the considered seeding level. This occasionally causes errors in the seeding accuracy, which subsequently impacts the final evaluation metric of cloud detection.

Segmentation Performance
Since existing cloud segmentation algorithms are designed for conventional LDR images, we evaluate them on the midexposure LDR images as well as the tonemapped HDR images. Our proposed HDRCloudSeg algorithm is the only one designed to make use of the full HDR radiance maps. However, for the sake of comparison, we also evaluate it for mid-exposure LDR and tonemapped images. The detailed evaluation results of HDRCloudSeg, along with the other cloud segmentation methods, are shown in Table 2. From Table 2, we observe that HDR imaging improves the cloud segmentation performance irrespective of the method used.

Methods
We observe that most of the benchmark algorithms (except for Li et al.) have a better performance with tonemapped HDR images as compared to the mid-exposure LDR image. This is because a tonemapped version exhibits fewer saturated pixels and clearer contrast between sky and cloud, as compared to the corresponding LDR image. However, our proposed method HDRCloudSeg using the entire HDR radiance map achieves the lowest error of 8.91% across all the methods.
Most of the other algorithms are biased towards a higher recall value (tendency to over-estimate cloud cover) with lower precision. These existing algorithms are based on a set of thresholds, either fixed or adaptive, and are therefore more prone to high error percentage. However, HDRCloudSeg uses the entire dynamic range of sky/cloud scenes to make a more informed decision in classifying a pixel as cloud or sky.
In this paper, we have proposed a novel approach to solve cloud segmentation in images captured by WSIs, using High Dynamic Range Imaging (HDRI) techniques. This greatly reduces post-processing steps (image inpainting and de-saturation), compared to sky camera designs based on a sun-blocker. These HDR images capture significantly more detail than traditional Low Dynamic Range (LDR) images, especially in the circumsolar region and near the horizon. We have shown that this method outperforms others. Our proposed methodology is reliable, efficient, and easy to be deploy.
In our future work, we plan to investigate how High Dynamic Range imaging improves the accuracy of other tasks, such as cloud classification (Dev et al., 2015a) or cloud height estimation (Savoy et al., 2015). In order to improve benchmarking, we will also work on expanding the sky/cloud HDR dataset introduced here with more images and distinctions between cloud types.

Database and Code
In the spirit of reproducible research (Vandewalle et al., 2009), we release the entire HDR sky/cloud image dataset as well as the source code of all simulations in this paper. The SHWIMSEG database is available at http://vintage.winklerbros.
Acknowledgements. This research is funded by the Defence Science and Technology Agency (DSTA), Singapore.