No Small Task: Improving Earth Engine’s Batch Processing Tools
By Julia Guo, Software Engineer & Sheba Rasson, Product Manager for Google Earth Engine
Batch tasks enable the processing of large geospatial datasets and complex analyses that would be impossible to perform in real-time due to time and memory constraints. Running multiple batch tasks concurrently allows you to maximize the computational power you get out of Earth Engine. We want to help you manage these tasks more effectively and with greater transparency and control. With this goal in mind, we are thrilled to announce a set of new features to help you better manage and prioritize your batch tasks in Earth Engine.
What’s new in task management?
Tasks are essential for exporting results, allowing you to transfer your processed Earth Engine data to Google Drive, Google Cloud Storage, or other destinations. The key new features focus on improving the efficiency and control of Earth Engine export tasks for paying commercial and operational users:
- Performance boost: Higher tier plans can run more export tasks concurrently, allowing you to get results faster.
- Customized control: Gain more granular control over your projects. Set custom concurrent task limits for your projects across a billing account.
- Task prioritization: Prioritize critical tasks within your Earth Engine project. Ensure your urgent analyses get preferential treatment for faster results.
Performance boost capabilities with commercial plans
For users tackling large-scale Earth Engine projects, our commercial plans unlock even greater flexibility and capacity. Higher commercial tiers offer increased limits on the number of tasks you can run concurrently. This means higher throughput!
Basic plans can run up to 8 concurrent batch export tasks per billing account, Professional can run up to 20, and Premium plans can run significantly more. To find the right commercial plan for your needs, please visit our Earth Engine pricing page.
Customized control for commercial customers
Commercial customers gain an even higher degree of flexibility with configurable project task limits. This allows you to tailor the maximum number of concurrent tasks per GCP project to align precisely with your workflow and analysis requirements. Enjoy fine-tuned control over your Earth Engine resources, optimizing efficiency and reducing potential bottlenecks.
If you’re interested in exploring configurable project task limits, check out the new Earth Engine CLI commands.
Task prioritization capabilities
Prioritize the analyses that matter most to you! With task prioritization, available for commercial Earth Engine projects, you can designate a custom priority to run higher priority tasks before those with lower priority. You can easily set task priorities at export time in both the Earth Engine Code Editor and the Javascript/Python client libraries.
What’s next for task management in Earth Engine?
We know how important it is to manage your tasks quickly and efficiently. As a result, we are investing in improving this experience by seamlessly integrating with the new Earth Engine page in Google Cloud console, launched earlier this year.
In the Earth Engine Cloud Console, you can find useful links to Earth Engine clients, datasets, and support resources such as the Earth Engine user guide. In the coming months, we’ll be building a new user interface for task management in the console. In this interface, you’ll be able to view and prioritize tasks with a few clicks and quickly connect with other components of the Cloud Console, including Billing and Monitoring.
