GISMODIS parable, code follow up
In a previous story I wrote about the levels of abstraction involved in remote sensing. For those interested in the methods of that project, in converting and organizing MODIS satellite data here’s what I’ve made, freely available on github. I’ll let the write up do the rest:
OC L2 netCDF Raster Extract
This product allows the quick and streamlined processing of NASA L2 Ocean Color data into standard raster types and projects. It can work with single or many files and allows simple visualization of the data. Metadata may also be extracted and files are customizably named and saved. This project was created for the purpose of processing many years of netCDF raster data for the general region of Patagonia but may function for any study region of the world. It sets up rasters so that they may later be easily retrievable and processed into higher level data products. This code could be adapted to work with other types of spatial netCDF files. Thanks to the rddj-template (https://github.com/grssnbchr/rddj-template) it is also completely reproducible and deployable with minimal modification.
Notes
This report was generated on 2017–11–28 11:39:42.
…
R-Script & data
The preprocessing and analysis of the data was conducted in the R project for statistical computing. The RMarkdown script used to generate this document and all the resulting data can be downloaded under this link. Through executing main.Rmd, the herein described process can be reproduced and this document can be generated. In the course of this, data from the folder ìnput will be processed and results will be written to output.
GitHub
The code for the herein described process can also be freely downloaded from https://github.com/mycid/rddj-template-TE-Raster-Extract.
The code for the project template may be found at https://github.com/grssnbchr/rddj-template.
Many thanks to Timo Grossenbacher @ (https://github.com/grssnbchr)
for creating this extremely elegant and useful template …
Project Purpose
This project was created to facilitate easy conversion of L2 netCDF rasters taken from NASA’s OceanColor Website run by the Ocean Biology Processing Group (OBPG) at NASA’s Goddard Space Flight Center (https://oceancolor.gsfc.nasa.gov/). Potential exists for others to easily adapt this script to handle L1 netCDF files as well or other large netCDF remote sensing files. The goals of the code are two-fold: 1. Convert many unprojected netCDF rasters to any raster file type in bulk. 2. Reproject and clip to the same study area many netCDF rasters with widely different swatch bounding boxes. An additional step has been to quickly visualize the results when necissary.
… ### Data description of input files Input files are netCDF files downloaded from the Ocean Color website as L2 products. These products are processed variables such as seasurface temperature and chlorophyll which are dervived from imperical relationships between irradiance and the physical variables. The l2 product is conveniently defined here: https://oceancolor.gsfc.nasa.gov/products/ netCDF files are structured to contain global attirbutes, band attributes and multiple bands. The metadata and file naming system is described here: https://oceancolor.gsfc.nasa.gov/docs/format/l2nc/ Example data for running this script includes 3 files containing chlorophyll alpha concentrations with associated variables and 3 files containing sea surface temperature with associated variables. Both files correspond to three images the study area contains on a single day, December 28, 2016.
Data description of output files
NASA_proj_metaD
Contains essential metadata extracted from the netcdf file global atributes. The bands extracted and the output raster file name are appended in columns at the end of the table. The metadata descriptive document may be found at: FILL IN
Output Raster files
Output raster files are generated and named using user specification of the function OCnetCDF2Raster. Raster files will be bounded by the specified study area and in the specified projection. Raster will not be scaled and cell values will contain the original values from the extracted raster. During rasterization and reprojection cells with multiple values from the netcdf file are averaged. NASA netCDF files are unprojected and use the WGS1984 latlon coordinate system. Each cell is given a centroid coordinate. Each raster has a resolution of aproximately 1km. This project uses UTM_S18 for the projection. Cells without data are NA. Users have the option of extracting the flagging layer contained within the netcdf file. For convenience, the flag values are described at the end of the document.
Output plots (optional)
Output plots can optionally be generated using the ExtractRaster or OCnetCDF2Raster function. Plots are saved as jpeg imaged and display the netCDF raster data in 10 discreet colored quantiles. The study area extent is also shown as a white polygon rectangle border. All points outside of the polygon are not included in the exported raster file but are shown here in the plot. Plots are titled by the date of the swath and saved with the same name as the original netcdf file included. …
Extract from single file and generate plot
#Set additional variables
output.proj <- “+init=epsg:5362”
date <- NetCDFdateT(nCDF)
plotname <- paste(band, date[1])
proj.ext <-
extent(250000,
xmax = 1200000,
ymin = 3700000,
ymax = 5500000)
#
Rasta <- ExtractRaster(
nCDF,
band = band,
Noflags = FALSE,
output.proj = output.proj,
proj.Ext = proj.ext,
makeplots = TRUE,
saveplots = TRUE,
plotname = plotname
)
## [1] “input//netCDF_files/A2016363183500.L2_LAC_OC.x.nc”
####Check output folder to verify results
Use wrapper function to extract netCDF rasters from multiple files
#Set stage for use of the netCDF2Raster function
inn <- list.dirs(path = “input/”)
files.oc <- list.files(path = inn, pattern = “OC\\.x\\.nc$”, full.names = TRUE)
files.sst <- list.files( pattern = “SST\\.x\\.nc$”, full.names = TRUE)
#list
chlor <- “chlor_a”
sst <- “sst”
out <- “output/”
proj.ext <-
extent(250000,
xmax = 1200000,
ymin = 3700000,
ymax = 5500000)
output.proj <- “+init=epsg:5362”
oc.netCDF.2raster(files.oc,
chlor,
filename = NA,
output.extension = “.tiff”,
output.proj = output.proj,
meta.name = “NASA_proj_metaD_Chlor.csv”,
proj.Ext =proj.ext,
meta.exp = TRUE,
Noflags = FALSE,
makeplots = TRUE,
saveplots = FALSE,
makeMeta = TRUE,
out = out)
## [1] “Extracting metadata…”
## Progress: 33% Progress: 67% Progress: 100% Done!
## [1] “Extracting Rasters…”
## [1] “input//netCDF_files/A2016363183500.L2_LAC_OC.x.nc”
## Progress: 33% [1] “input//netCDF_files/A2016363184000.L2_LAC_OC.x.nc”
## Progress: 67% [1] “input//netCDF_files/A2016363201500.L2_LAC_OC.x.nc”
## Progress: 100% Done!
## user system elapsed
## -96.38 -24.34 -165.86
Running the RMD
First note this Rmd is principally designed to be reproducible. It will download older versions of packages to achieve this result, a function that will not be strictly necessary given the methods involved. If you would like to avoid the use of checkpoint or other such measures just use the functions contained in the scripts.
The main.Rmd file will run and build an html file when knitted. No input is needed. To use the Rmd file as a tool simply add your own data to the input folder and replace function variables to match your desired output.
Using the scripts
The scripts provide fully ready to use functions but require a number of spatial data packages that will need to be download first. The main.Rmd file automatically does this, but if you would like to use just the script functions open the script to see what is required. Use the main.Rmd file as an example for using the script functions.
Data Products
nice rasters of your file types, a metadata spreadsheet, and images that look like this:
Aknowledgements
Special thanks to Timo Grossenbacher (https://github.com/grssnbchr) for building the RMD template and to the Huinay Foundation for providing the project and assistance. http://www.huinay.cl/site/sp/
Author
- Trevor Eakes
