Clip and Ship: Batch Clips using Planet’s Clips API
Over the last couple of months Planet has been working hard on implementing the Education and Research Program allowing researchers to gain access to datasets for non-commercial research purposes. The accounts have an quota of 10,000 square kilometers of data per month. However, only the area downloaded is countered against the quota.
The clips API is offered as a compute API meaning the clip function is an additional computation performed on an asset. Based on their documentation the clips API allows cropping a raster to a user specified geometry. The API then packages the results along with the modified metadata files into a zip file per clip. Since this operation is called on demand there is a response time after which a download token is generated allowing the user to download the image. The Clips API endpoint can be found here you can log in using your Planet Explorer credentials.
Additional details on the Clips API can be found here and a change-log is available to make sure you don’t miss out on any updates. Since this feature is still Beta the number of assets you items and assets you can clip using this approach is limited and is listed as below.
The current batch tool I created borrows components from my earlier projects on Planet Earth Engine Pipeline CLI . This includes tools to store your Planet API key also to be used with the Clips API along with activating assets for download. The tutorial introduces the Command Line Interface designed around this to facilitate batch processing and downloading of clipped area. The idea is to provide an easy mechanism for users to clip multiple AOI(s) as needed and save on their area quota.
Housekeeping and Pre-Installs
The program requires bare essentials apart from native python installed on your system. Incase a few of you are on a windows machine and based on discussions with a few users who are using ArcMap you might have to configure and add this to your system path. There is an old tutorial which helps you do this: just change the ArcMap version and you should be able to go to command prompt and start python by typing python. You need a planet account which gives you access to your very own API key. (The free
account gives you access to California data under a CC BY-SA 4.0
license) and you can also apply for the Education and Research Account.
- To finally install you can go to the GitHub page at Planet-GEE-Pipeline-CLI. As always two of my favorite operating systems are Windows and Linux, and to install on Linux
git clone https://github.com/samapriya/Clip-Ship-Planet-CLI.git
cd Clip-Ship-Planet-CLI && sudo python setup.py install
pip install -r requirements.txt - For windows download the zip and after extraction go to the folder containing
“setup.py”and open command prompt at that location and typepython setup.py install
pip install -r requirements.txt
You can use the tool without any installation by simply going to the pclip folder and typing python pclip.py. If you have installed everything and your dependencies are met you should be able to typepclip -h
in command prompt and get a list of all the tools. For the first run it will ask you to enter your planet API key since it uses that to perform any action.
Save your Planet API Key
The tool is designed to save your API key so that you don’t have to do this again and again, a new users API key overrides yours.To set a planet key enter the key when asked for password
pclip planetkey
To create your area of interest which would be used to clip the image assets and to structure it based on date and cloud filters follow instructions from my earlier post.(Just replace ppipe by pclip).
So to structure your geojson file into a json file with filters you need to runpclip.py aoijson --start “2017–06–01” --end “2017–06–30” --cloud “0.15” --inputfile “GJSON” --geo “local path to map.geojson file” --loc “path where aoi.json output file will be created”
Getting Started : Activate and List
This assumes that you have downloaded an area of interest and created a structured json based on date and cloud filters. Since you can only clip datasets that are activated I modified the python client to generate item ID list post activation. So the sequence of steps would include
pclip activate --aoi "local path where you created aoi.json file" --action "activate" --asset "PSOrthoTile analytic"
Remember the type of item and asset you activated since you will be using the same types for your clip functions.
Listing or generating an idlist is the next step and all you need you do for this
pclip idlist --aoi “local path where you create aoi.json file” --asset “PSOrthoTile analytic”
Submit Clipping Request
The next step is to submit the clipping request using the item id, item type and asset type we used . To do this we can use either the jsonc or the geojsonc command depending on whether we using json or geojson files for clipping. By using the json we used to create the idlist we would have the following setup.
pclip jsonc --path “local path where you create aoi.json file” --item “PSOrthoTile” --asset “analytic"
Download Clipped Requests
Once the clip requests have been submitted to the compute process it might take some time for these to complete after which we can execute the download request. The setup for the same includes only an output directory where the zipped files are to be downloaded.
pclip downloadclips --dir "Local directory where you want your clipped images to be saved"
If the clips have not yet been processed you will get "running" as the output instead of downloading and you can just retry the same command again in sometime.
Extra Goodies for this post includes the capability to unzip and sort all images and metadata into separate folders. Image and Metadata Sub-folders are created within the location where you want your files to be unzipped and moved into respective folders for ease of use.
Sorting Files
The last step would include unzipping and sorting images and metadata xml files generated from the clip request. The setup includes the parent directory containing the zipped files and a local directory where the files are to be unzipped, staged and sorted. This should take minimal time and uses the standard zip library already included with native python. The command also deletes your zipped files to clear up space.
pclip sort --zipped “local path where zipped files were downloaded” --unzipped “local paths where unzipped files are to be sorted”
There you have it, you can now download only your area of interest instead of downloading entire scenes and hence save on your area quota. The tool will only download assets that are activated so don’t forget to activate assets before sending in the clip request. This project was based on discussions with Planet’s Education & Research users and I hope it helps save up on your allocation quotas as you keep working on some amazing topics.
For now I think this should work and future work includes a multiprocessing module based download and sorting client to speed up the entire process even further as request sizes get larger. I am including the citation to help me justify building these tools while I work on my PhD.
You can cite the tool using
- Samapriya Roy. (2017, September 27). samapriya/Clip-Ship-Planet-CLI: Clip-Ship-Batch Planet Command Line Interface. Zenodo. http://doi.org/10.5281/zenodo.998031
You can cite the Planet API using:
- Planet Team (2017). Planet Application Program Interface: In Space for Life on Earth. San Francisco, CA. https://api.planet.com.
Thanks to discussions with Joseph Mascaro, Trevor McDonald for following up when the conversation got started. For more questions about this project, and for further discussion, you can email me. You can also always reach out to the Planet support team at support@planet.com.
