Draft 30 Apr 2020, Andrew Sayer
This algorithm defines whether a given pixel is likely to be partially or fully covered by a cloud ("cloudy") or not ("cloud-free" or "clear"). This is a basic input to other elements of a satellite data processing chain: most algorithms that derive atmospheric or surface properties either deal with only cloud-free cases, or only cloudy cases. While some algorithms have their own internal cloud mask; this product constitutes a cloud mask which can be used stand-alone or as an input to those downstream algorithms. It has two main outputs for each pixel processed:
The algorithm is based on MODIS heritage, and the implementation so far is on MODIS data as a proxy for the PACE OCI. It can in principle be applied to other sensors with sufficiently similar spectral bands (e.g. VIIRS).
Satellite instruments do not measure cloudiness directly, but rather reflected or emitted radiation at the top of atmosphere (TOA). The algorithm implements a number of spectral and spatial tests on the signal observed at TOA in order to determine whether or not a given pixel is covered by cloud or not. In the current implementation, these tests are drawn from the subset of tests applied in the NASA MODIS and VIIRS cloud products using features of those sensors which will be available to PACE OCI. These tests are based on the expected spectral and spatial characteristics of cloudy scenes (e.g. Ackerman et al., 1998, Frey et al., 2008). Some of the tests (known as "clear-sky restoral" tests) are to identify non-cloud features which may trigger false positives in other tests (e.g. heavy atmospheric aerosol particle loading). The main difference between the MODIS/VIIRS implementations and OCI is that OCI will not have thermal infrared bands and so can rely on only solar band tests. In the future, this algorithm may be enhanced to make use of some features of OCI (e.g. O2 absorption bands) which are not present on MODIS and VIIRS and have demonstrated utility for cloud masking (Mei et al., 2017).
While Ackerman et al. (1998) and Frey et al. (2008) describe the physical basis behind these tests, the specific implementation logic is described in Ackerman et al. (2010) and Frey et al. (2019) and adapted from there. Pixels are only processed if they are considered daytime (solar zenith angle smaller than 85°). The algorithm first assigns a category to each suitable pixel, using a combination of ancillary data and geometric/radiometric factors. The potential classifications are water, coastal, land, or desert. Additional spectral tests are applied to pixels identified as coastal or land (but not desert) to determine whether they are covered by snow (Klein et al., 1998). In addition, the glint angle (difference between the scattering angle and the angle of specular reflection) is calculated for pixels classified as water, and the Normalized Difference Vegetation Index (NDVI) is calculated for all pixels. These classifications determine thresholds and processing logic used in subsequent tests. As well as tests to identify clouds, there are several so-called ‘clear sky restoral’ tests used to un-flag pixels which may have generated false positives from other cloud mask tests.
Following the nomenclature of Ackerman et al. (2010) and Frey et al. (2019), the specific tests described therein implemented for the PACE proxy are:
The three cloud detection tests (solar cirrus, visible reflectance, visible ratio) are probabilistic and include lower and upper thresholds on the quantities tested. Specifically, for an observation with value β used in a test with lower and upper thresholds α, γ, the confidence level $F_{i}$ returned by test i that the pixel is clear is:
Fi={
0: β < α
1: β > γ
(β-α) / (γ-α): α < β < γ
}
Ackerman et al. (2010) define $F_{i}$ directly as the probability of a clear-sky pixel according to that test. The three tests (after any applicable restoral checks) are then combined to determine an overall cloud mask confidence Q. Ackerman et al. (2010) note that certain tests are more sensitive to certain types of clouds so a multiplication of $F_{i}$ for all tests to obtain Q may be inappropriate. As a result, they first group tests according to the features they are most suited to detecting to get an effective $F_{group}$ for each group. As the MODIS/VIIRS thermal tests are not relevant for OCI, the OCI tests involve cloud mask tests from only their groups III (visible reflectance, visible ratio) and IV (solar cirrus). Thus, in the present implementation, $F_{III}=min(F_{visret}$,$F_{visrat}$) and $F_{IV}=F_{cirrus}$ (where subscripted visret, visat, and cirrus refer to the relevant cloud tests).
Then, Q is obtained by multiplying $F_{i}$ from all applicable groups of tests for that pixel and taking its Nth root (where N is the number of groups of tests applicable for that pixel). Here, $Q=\sqrt{F_{III}F_{IV}}$ as in most cases tests from both groups III and IV are applied to each pixel. Note that this approach is clear-sky conservative, as if $F_{i}$=0 for any test, then Q=0.
The final categorical cloud mask output is defined as follows:
Q={
≤ 0.66: Confidently cloudy
0.66 ≤ 0.95: Probably cloudy
0.95 ≤ 0.99: Probably clear
> 0.99: Confidently clear
}
The value of Q is also provided on a per-pixel basis, in case users wish to apply their own categorisation.
The binary cloud mask output is set to cloudy for each valid pixel which is either "confidently cloudy" or "probably cloudy", or is adjacent (in the satellite swath) to one or more pixels with those classifications. Other valid pixels are marked as clear. A valid pixel is a pixel for which one or more of the cloudiness tests was successfully applied.
TBD
The cloud mask will be validated against available coincident ground-based, airborne, and/or spaceborne lidar and radar, data. This has been the approach taken with e.g. MODIS (Ackerman et al., 2008, Wang et al., 2016). The performance of both the "confident" classes, as well as the characteristics of the "probably" classes, will be examined. The main relevant metrics for the "confident" classes are the fractions of true/false positives and true/false negatives as compared to these reference data sources. As these have limited spatiotemporal availability, comparisons of cloud detection frequency with other satellite imagers will also be performed.
Ackerman, S. A., Strabala, K. I., Menzel, W. P., Frey, R. A., Moeller, C. C., and Gumley, L. E. (1998), Discriminating clear sky from clouds with MODIS, J. Geophys. Res., 103 (D24), 32141-32157, doi: 10.1029/1998JD200032
Ackerman, S.A., R.E. Holz, R. Frey, E.W. Eloranta, B.C. Maddux, and M. McGill (2008), Cloud Detection with MODIS. Part II: Validation. J. Atmos. Oceanic Technol., 25, 1073-1086, doi: 10.1175/2007JTECHA1053.1"
Ackerman, S. A. et al (2010), Discriminating clear-sky from cloud with MODIS algorithm theoretical basis document (MOD35), Version 6.1, available online at doi: https://atmosphere-imager.gsfc.nasa.gov/sites/default/files/ModAtmo/MOD35_ATBD_Collection6_0.pdf
Chen, S. and T. Zhang (2015), An improved cloud masking algorithm for MODIS ocean colour data processing, Remote Sens. Letters, 6 (3), 218-227, doi: 10.1080/2150704X.2015.1026955
Frey, R.A., S.A. Ackerman, Y. Liu, K.I. Strabala, H. Zhang, J.R. Key, and X. Wang (2008), Cloud Detection with MODIS. Part I: Improvements in the MODIS Cloud Mask for Collection 5, J. Atmos. Oceanic Technol., 25, 1057–1072, doi: 10.1175/2008JTECHA1052.1
Frey, R. et al (2019), The Continuity MODIS-VIIRS Cloud Mask (MVCM) User Guide, Version 1,9 Product User Guide – Version 1.0, available online at doi: https://ladsweb.modaps.eosdis.nasa.gov/missions-and-measurements/viirs/MODIS_VIIRS_Cloud-Mask_UG_Feb_2019.pdf
Klein, A.G., Hall, D.K. and Riggs, G.A. (1998), Improving snow cover mapping in forests through the use of a canopy reflectance model. Hydrol. Process., 12: 1723-1744. doi: 10.1002/(SICI)1099-1085(199808/09)12:10/11<1723::AID-HYP691>3.0.CO;2-2
Mei, L., M. Vountas, L. Gómez-Chova, V. Rozanov, M. Jäger, W. Lotz, J. P. Burrows, and R. Hollmann (2017), A Cloud masking algorithm for the XBAER aerosol retrieval using MERIS data, Remote Sens. Environ., 197, 141-160, doi: 10.1016/j.rse.2016.11.016
Wang, T., Fetzer, E. J., Wong, S., Kahn, B. H., and Yue, Q. (2016), Validation of MODIS cloud mask and multilayer flag using CloudSat-CALIPSO cloud profiles and a cross‐reference of their cloud classifications, J. Geophys. Res. Atmos., 121, 11620-11635, doi: 10.1002/2016JD025239
Sample data products are available on request.
