Landsat 8 Cloud Free Composite created in Google Earth Engine

Google Earth Engine Asset Manager and Addons: Building Tools of the Trade

In the last couple of years Google Earth Engine has transformed the way we look at doing remote sensing and geospatial analysis. It has allowed for users to truly move to a cloud native experience where all they need to run an analysis is a browser and an internet connection. The way in which this works has been detailed recently in a paper “Google Earth Engine: Planetary-scale geospatial analysis for everyone”. Apart from having a Python Client and a Javascript interface the Earth Engine Python API allows you to interact directly with your account and perform asset and task level functions.

Python Earth Engine Command Line Interface v0.1.123 with Python 2.7.14

The tools allow you to mostly perform single operation command line interactions with some recursive functions built in (for example:)

earthengine task cancel all or earthengine ls “users/userid/folder" -r

Setting Up Google Earth Engine Python API

For now no matter what operating system you are using as long as you have python 2.7 you should be able to install the EE Python API before which make sure you have a Google Earth Engine Account. If you don’t have a developer account sign up for one here and make sure you follow the instructions to install the python API. The API gets updated frequently and as does the install process as needed.

The motivation for this project came from the discussions on Batch Uploading Project that had been beautifully implemented by Lukasz. I wanted to build more tools around it apart from the upload and delete tool and hence the process started as a project to build addons to the existing CLI. This includes the upload and recursive delete tool from the existing project and additional tools to interact with the native EE CLI.

Install GEE Asset Manager with Addons

  • To finally install you can go to the GitHub page at GEE Asset Manager with Addons
    git clone https://github.com/samapriya/gee_asset_manager_addon.git
     cd gee_asset_manager_addon && sudo python setup.py install
     pip install -r requirements.txt
  • for windows download the zip and after extraction go to the folder containing “setup.py” and open command prompt at that location and type
    python setup.py install
     pip install -r requirements.txt

You can use the tool without any installation by simply going to the ppipe folder and typing python geeadd.py If you have installed everything and your dependencies are met you should be able to type

geeadd-h

in command prompt and get a list of all the tools.

— — — — — — — — — — — — — — — — — — — — — — — — — — — — — — —

Tool Scopes

Overall- Tools that are applied to overall account or assets

Recursive- Tools that process all assets within folders or collections

— — — — — — — — — — — — — — — — — — — — — — — — — — — — — — —

Google Earth Engine Asset Manager and Addons

The Google Earth Engine Batch Asset Manager and Addon was started as an extension to just the upload feature and were mostly inspired by things I wanted to do or was stuck finding solutions to over a period of time. This tool has had many iterations to include general improvements and additional functionalities which are discussed below with some examples.

Google Earth Engine Addon CLI
Earth Engine User(Overall)

When a new user uses the Earth Engine CLI(EE CLI)they have to use an account generated key which helps the CLI identify the google account linked to all functions. It is generally brought about by using earthengine authenticate and this is included as a feature within the tool called ee_user. It benefits from using a clipboard function which copies the url link incase the computer being used does not support a browser operation and the link can be used in a different computer and the authentication key can be imported back.

Some of the tools were built to basically replicate the EE CLI so that the user does not have to switch between using multiple CLI(s). (This includes the create tool which is equivalent to built in create tool)

List Function(Recursive)

The list function or geeadd lst function allows you to recursively list and print all components in a folder or collection and either print them or export them as a text file. This is an extension of the ls function and allows the user to export the results.

List Tool GEE Addon CLI
EE Report(Overall)

This is one of the recent tools that was added to the addons CLI and this allows the user to print a report of all assets associated with your Earth Engine account. It includes the following information

[Type,Full Path,No of Assets, Size,unit(MB),Owner,Readers,Writers]

Earth Engine Report Tool GEE Addon CLI

The tool takes the complete path of the CSV file where the report is written out. Depending on the number of asset the time take varies but it is now capable of handling all assets including collections, images and tables.

Asset Sizes(Recursive)

This tool was created to generate the overall size of an any EE assets[Images, Image Collection,Tables,Folders] so that we get a more familiar output in terms of asset size in (MB, GB, TB) as sizes vary. Current implementation of this is the du function in earth engine working only for ImageCollections and Folders and the results are in bytes.

AssetSize Tool GEE Addon CLI
Tasks and Task Report(Overall)

Tasks is a simple tool to allow you to look at all task states and values so we get a read out of Completed , Running, Ready, Failed and Cancelled. The taskreport tool is incase you want to look at details such as task id, state, reason for failure, output links among other things and generates a csv file with task details.

Tasks and Task Report in GEE Addon CLI
Move and Copy Assets(Recursive)

Once in a while you will have the need to rename a collection or folder path but EE does not allow you to do that without handling assets which reside inside the parent collection or folder. An easier way to do this rename is to use the move assets tool to move the collection to a new collection or assets from one folder to the other. This will also allow you to copy assets to which you currently have read access only to make sure you can write to it directly. This is a recursive function meaning it can move or copy entire collections and all assets within collections.

Move and Copy tools GEE Addon CLI
Delete and Cancel(Recursive)

These two tools are post processing tools used to either delete collections iteratively or cancel all running tasks. Incase you want yo cancel all tasks whether running or ready by using earthengine task cancel all . This tool only cancels running tasks and not ready tasks.

Delete and Cancel Running Task Tool GEE Addon CLI
Collection Access and Collection Property(Recursive)

These are probably one of the most useful tools that I had to build into the addon. It turns out that changing read and write permissions to simply a folder does not change permission for all assets within the folder which have to applied individually. The access tool allows you to iteratively apply permissions change and add users to all assets within folder and/or change permissions to individual collections or images. The collprop tool allows the users to set description , labels and tags for their assets similar to native collections which is currently not available.

Access and Collection Property Setup Tools GEE Addon CLI

As the next generation of analysis tools become available Google Earth Engine keeps evolving to allows users to perform massively complex analysis and asking bigger and better questions. For now I will keep working on improving and adding new tools to the GEE addon.

Future Updates includes creating an autosuggest terminal for the addon and wrapping a GUI (Experimental for now)
Autosuggest for GEE Addon CLI

You can cite the Google Earth Engine Addon Tool using

and you can cite the Google Earth Engine API

  • Noel Gorelick, Matt Hancher, Mike Dixon, Simon Ilyushchenko, David Thau, Rebecca Moore, Google Earth Engine: Planetary-scale geospatial analysis for everyone, In Remote Sensing of Environment, 2017, , ISSN 0034–4257, https://doi.org/10.1016/j.rse.2017.06.031.

Thanks to all within the Earth Engine Community for ideas and insights into problem solving and discussions. You can log issues with the tool here and I can try to work on it when possible.