Using Google Earth Engine with Julia
Cloud-based geospatial processing with a modern state-of-the-art programming language
Introduction
Google Earth Engine is a powerful cloud-based geoprocessing platform that allows users access to petabytes of earth observation and model data with collocated compute. The Earth Engine (EE) platform is an invaluable resource for Earth science research and applications and has revolutionized the Earth science/geospatial field due to it ease of use and allowing users to quickly prototype algorithms and produce outputs at planetary scale. To interface with the platform users develop workflows using either a JavaScript and/or Python API. By having only a JavaScript or Python API limits some developers to those languages. However, projects (like rgee) have emerged to allow developers to interface with EE in their favorite languages.
Julia is a modern dynamic, high-level programming language, for scientific and numerical computing. It has the feel of a scripting language with the performance compiled languages (thanks to its JIT compilation). It is a programming language full of features for technical computing and is gaining traction for scientific computing.
This blog post walks though setting up the Julia environment for interfacing with EE API. Additionally, examples are provided to highlight syntax and capabilities. Lastly, new EE features are shown together with Julia to highlight where the real benefit of using the two together can be.
Disclaimer: I have written this article as part of my process in learning Julia. This is meant to be an example for others and a reference for myself. I do not focus on comparing different languages as all languages have their own inherent merits. I hope others find it useful. All opinions in this article are my own.
Setting up the environment
To use Earth Engine (EE) with Julia, we will use the existing Python API and call the functions using Julia. This is done through Julia’s PyCall package but we will need to install the EE API for use within the Julia environment (and a few other google cloud packages for later on):
using Pkg;
Pkg.add("PyCall");
Pkg.add("Conda");
Pkg.add("Plots"); # used for plotting
Pkg.add("Colors");
Pkg.add("FileIO");
Pkg.add("ImageIO");
Pkg.add("Images");using Conda;
Conda.add("earthengine-api",channel="conda-forge");
Conda.add("google-api-python-client",channel="conda-forge");
Conda.add("google-auth-httplib2",channel="conda-forge")
After we have the environment set up, we can import the package, authenticate, and see if the we can initialize an EE session.
using PyCall;
ee = pyimport("ee");
ee.Authenticate(); # authentication is a one-time process
ee.Initialize();If all goes well and there are no errors when importing the EE package and initializing a session 🤞, then EE can be used with Julia!
EE+Julia Examples
To get started illustrate how to execute EE workflows using Julia, some examples using the Python API are replicated using Julia syntax.
Hello EE!
The first example is focused on importing the packing and performing a small geospatial process. Here the SRTM elevation data is imported and queried at the geospatial coordinates of Mount Everest to get the elevation value. This uses an ee.Image and ee.Geometry to make sure everything is working correctly.
As seen, the syntax is almost identical to using the Python API aside from small Julia changes. Specifically, Julia uses double quotes to define strings most of the time and trying to use single quotations will throw errors. Additionally, when defined the geometry variable xy there is an ellipsis after the coordinate list; this is splatting where one argument is split apart into many different arguments for a function with unknown arguments. Again, the syntax is largely the same because we are interfacing with the Python EE package through Julia. More in-depth examples will highlight more Julia syntax.
Sampling data from an image
As a more extensive example, we will sample data from a raster dataset. This is a common workflow for geospatial sciences whether looking at relationships between variables or sampling data for ML workflows. Here we load in Landsat image, sample band values, and plot the relationship of the bands.
This example ends up loading the data as Julia arrays, this is really helpful if you want to leverage Julia’s powerful processing capabilities. This also highlights using the Julia Plots package for plotting.
Image processing and visualization
The true value of EE lies in its capability to process large amounts of raster (Earth Observation) data in space and time. This next example is a common workflow to calculate NDVI from a Landsat 5 image. For this example we use Julia to define a function, apply it to the image, and pull the results for visualization.
As we can see from the example, we can access an image, apply a function on the image and visualize the results. We use Julia to handle a lot of the nitty-gritty details of getting a colormap in hex color code with ease allowing users to focus more on the science. These concepts can be abstracted to apply over EE image collections and get results.Also, there is a whole lot more EE functionality that we can access using Julia via the Python API but this would be a very long article if I were to highlight everything.
The code for all of these examples can be accessed here: https://gist.github.com/KMarkert/aa90601ca588b97a507df5a056e73892
Why use Julia to interface with EE?
This may be going on in your head, “why use Julia if you are simply calling the Python API?” This is a fair question since this is just interfacing with the Python EE API using there would be minimal performance gains. One answer is user preference in programming languages. However, most of the advantages of Julia come from working earth observation/remote sensing data outside of EE. The Earth Engine team recently released a cloud REST API for interacting with EE that allows user to access data either directly from the EE data catalog or after EE has performed some computations! This is huge as it allows processes to be chained together leveraging EE to handle the big computations on EO data and use Julia for ML, modeling, or whatever your heart desires!!!
Julia and the EE REST API
🚨Setup required!🚨 Before going through the REST API examples, please make sure you setup your cloud project and get a service account secret key to access the REST API. Details on how to do this can be found here.
To start off, we will setup the environment and see if we can request information on the Landsat collection. First, we load in the Python Google authentication packages that we installed earlier (not sure if there are Julia packages available to do this), then we authenticate a session using our credentials, and request information from a URL endpoint that corresponds to the Landsat.
using PyCall, JSON;service_account = pyimport("google.oauth2.service_account");
gauth = pyimport("google.auth.transport.requests");SECRET_KEY = "<PATH_TO_SECRET_KEY>"credentials = service_account.Credentials.from_service_account_file(SECRET_KEY);
scoped_credentials = credentials.with_scopes(
["https://www.googleapis.com/auth/cloud-platform"]);session = gauth.AuthorizedSession(scoped_credentials);url = "https://earthengine.googleapis.com/v1alpha/projects/earthengine-public/assets/LANDSAT";
response = session.get(url);
asset = JSON.parse(response.content)
If all goes well from the code above the contents of the variable asset should look like this:
Dict{String,Any} with 3 entries:
"name" => "projects/earthengine-public/assets/LANDSAT"
"id" => "LANDSAT"
"type" => "FOLDER"This means we are able to successfully authenticate to use the EE REST API and get results 🎉! It is nice to be able to access the EE catalog metadata using simple HTTP requests but there is more…now we can chain together more extensive workflows to access raw data and read into Julia as arrays.
Requesting data
To access data directly from EE we create a session to send HTTP requests to EE’s REST API, define some parameters such as the asset to query and spatial domain to request data, then send the request and read as a Julia array. Once we have the raw data as a Julia array, we can leverage the high performance of Julia for additional processing.
Link to script: https://gist.github.com/KMarkert/055a9334d64ad8e2d9e1d6b146f5e5a0
This was a straightforward example on accessing data from EE using the REST API and visualizing the image. However, there are a whole lot more capabilities available now using the REST API particularly with serializing an expression graph and getting the results. Retrieving large amounts of data from EE can be cumbersome and Julia will be able to handle the data and process with ease. Imagine whatever Earth science based process that you wished EE could do and now do it. This is something I am particularly excited about as it allows for more flexibility in working with EE!
Concluding remarks
Julia is a modern programming language that focuses on performance for technical computing whereas Earth Engine is a cloud-based geoprocessing platform that allows access to petabytes of Earth science data. Using the two together, we can leverage powerful geospatial processing with expressive syntax. Moreover, we can access data directly or after some computations from Earth Engine and work with the results directly in Julia. While the concepts of interfacing with EE and using the REST API are transferable to other languages, I foresee the EE+Julia being a formidable combination 💪 in the future of geospatial computing and Earth science applications (this is my opinion). I say this due to the straightforward and expressive syntax that Julia provides and powerful compute that the EE platform and Julia both bring to the table.
I plan on using EE with Julia and exploring more on how to access compute results and pass them into external processes like ML workflows or hydrology models using Julia. I hope to see Julia’s use with EE and in the broader Earth science field!
I would lastly like to thank the
team for everything they do and revolutionizing the Earth science field with the platform.Circling back to my earlier point of packages in other languages to interface with EE, I would love to see an EE.jl package (or something like that) and if there is interest in planning/implementing something like this, please get in contact with me via Twitter or on GitHub.
