Harness the power of Sentinel Hub, xcube, EOxHub, GeoDB and more in Euro Data Cube
Start with simple Jupyter Notebooks then scale it up
You may have come across the Euro Data Cube when reading posts in the Sentinel Hub Blog or in the news from the European Space Agency (ESA). But what is Euro Data Cube exactly, and why does it matter to the Earth Observation(EO) industry? In this blog post, we will take a closer look at this online platform and the main features that make it the EO information factory of the future.
Every day, petabytes of EO data are acquired by satellites, providing us essential information about the Earth’s physical, chemical and biological systems that can be leveraged for decision-making in a variety of fields, from agriculture to understanding the impacts of a pandemic. However, getting access to EO data and preparing them to analysis-ready format can be time or resource consuming and present a barrier to extracting useful information from EO data.
Euro Data Cube has set out to lower these technical barriers by bringing together already operational data access web services and processing tools in a single platform. The portfolio of services includes a managed workspace environment to deploy applications and run algorithms, a cloud API to access global satellite imagery and a PostgreSQL database to store and manage geospatial feature data as well as processing tools including a powerful xarray based Python library to exploit EO data.
The unique combination of services makes it convenient for beginners or EO experts alike to (1) easily set up processing workflows, without the need of installing software packages (2) access EO data in a variety of formats tailored to the application without having to download individual image scenes or perform pre-processing steps before analysis and (3) promote and offer solutions to the world through a managed marketplace.
What is a data cube?
A data cube is a multi-dimensional data array. In the EO domain, a data cube serves satellite imagery as a multi-dimensional array, typically with two spatial dimensions, one temporal dimension and an arbitrary number of attributes such as bands or indices. The georeferencing of satellite images guarantees that the pixel location stays consistent between different layers of the data cube. The goal of a data cube is to serve users with analysis-ready data, thus reducing the burden of pre-processing and data preparation.
How to get started with Euro Data Cube
To get started, you will first need to simply sign up for a Euro Data Cube user account. Once logged in, you have access to your personalised dashboard, which is your central place to keep an overview of your account’s activities, workspace, service subscriptions and data products. All Euro Data Cube services are found in a single harmonized marketplace interface. From here, choose a service that suits your needs, then proceed to activate the service accordingly and start using it immediately. Most of the offering has a free trial option, make sure to make use of it!
EOxHub workspace: Managed compute and storage environment
The easiest way to get into data processing and analysis is probably to get your own hosted cloud EOxHub workspace which provides you with all the necessary computational and storage resources to run your Jupyter Notebooks within a managed JupyterLab or host applications. When you deploy JupyterLab, it comes with all dependencies preinstalled and the necessary credentials of all active services prepared as environment variables. Therefore, running a script simply requires you to call the existing environment variables at the beginning of your Notebook. No need for setting up credentials manually.
Sentinel Hub: On-the-fly data access
Whether you are looking to gain access to open-access satellite data (Sentinel, Landsat, MODIS) or VHR satellite data from commercial missions, your needs are covered by Sentinel Hub. Currently, the commercial data supported are PlanetScope, Pleiades and SPOT and these can be imported to Sentinel Hub via third party data import API. Browse through the full list of the available data for more detailed information.
Sentinel Hub’s Processing and OGC compliant APIs take care of all the complexity of archiving, processing and distribution and support on-the-fly processing of satellite imagery, making it easy to retrieve satellite data over any AOI and specific time range.
Sentinel Hub’s Processing API has been integrated into Euro Data Cube providing you with direct access to full archives of satellite data directly usable in your workflow. Custom scripting is supported, and this gives you the flexibility to define simple JavaScript code to process and return the data that you need. Besides, sentinelhub-py has put together handy tools for requesting satellite imagery from within Python scripts.
As you may already know, there are other functionalities offered by Sentinel Hub. These functionalities are automatically included in your Sentinel Hub subscription with no extra cost. They include Bring your Own Data, Batch Processing API for processing large scale data (such as cloudless mosaics) as well as the Requests Builder, an application with a friendly user interface to quickly create Sentinel Hub API requests.
xcube: Data cube toolkit based on xarray, dask and Zarr
For more advanced operations, xcube, a Python-based solution to transform and store input data into a data cube has been integrated into Euro Data Cube. xcube is an open-source Python package for generating and exploiting data cubes powered by xarray, Dask and Zarr. To extend xcube functionalities to support generating data cubes from specific data sources, several plugins, for example, xcube_sh, have been developed for that purpose.
xcube_sh is the xcube plugin developed to generate data cubes on-the-fly from Sentinel Hub API. To use it, all you need is a valid Sentinel Hub subscription. No need to download and install the xcube packages locally, as it has been integrated into Euro Data Cube and comes automatically installed in the JupyterLab environment.
The first step of the xcube processing model is to configure a data cube (also called xcube dataset), with the details of the data that you need to fetch. An xcube dataset is simply an xarray Dataset instance structured such that it contains one or more data variables whose values are stored in cells of a common multi-dimensional, Spatio-temporal grid. It is self-describing, as it contains metadata defining the cube and all its variables.
The next step is to open the xcube dataset. When an xcube dataset is opened, only its structure and metadata are loaded into memory but no request is made to fetch the data. The actual data is requested from Sentinel Hub on demand, only when a process needs it. If new data is generated, it is added to the xcube dataset and can be fetched when needed. This allows for parallel and out-of-core computation while saving resources at the same time. Explore Sentinel Hub-xcube integration notebook to get started with manipulating xcube datasets.
The xcube generator has also been integrated into Euro Data Cube. It provides a simple graphical user interface to generate persistent data cubes from various data sources (ESA CCI Open Data Portal, Sentinel Hub API and Climate Data Store API) and store them in your S3 bucket. The generated data cubes can then be visualised with the integrated xcube viewer or imported to Jupyter Notebook for analysis as outlined in this how to use xcube Generator guide.
GeoDB: Store and exploit feature data
Euro Data Cube provides a service (GeoDB)for storing and manipulating feature data too. It is common for EO analysis workflows to include feature data, therefore a feature database becomes essential. GeoDB, a database plugin for xcube, fully supports reading features into GeoDataFrames and storing them in a custom PostgreSQL database. Features in the database can be queried via a Python client and are returned as GeoDataFrames that offer many ways of handling or displaying your feature data. Start manipulating feature datasets with this getting started notebooks or explore Land Parcel Identification System spatial data already available in the database with this tutorial notebook.
Collaborate and do business
Collaboration is beyond doubt one way to widen the impact of EO on society. With Euro Data Cube, you not only get to use our data, tools and cloud workspace but also get a prime space to contribute and sell your solutions to the world on the dedicated marketplace. The marketplace enables you to onboard your solutions and thus share your know-how with the EO community. As you focus on maintenance of your contributions, the advertising, customer management and billing as well as credentials management for products and services you offer on the marketplace are managed for you by us. Simply browse through our documentation for guidelines on how to get started with onboarding your solutions to the marketplace.
Using Euro Data Cube for a good cause
Much of the economic and social activities have been cut down due to COVID-19 crisis. This has impacted industries such as agriculture, shipping, transport and aviation. Analysis of EO data provides valuable insights into assessing these impacts and helps organisations and the general public to respond accordingly.
To support COVID-19 impact assessment, Euro Data Cube recently organised a custom script challenge and offered Euro Data Cube services for free to participants to develop innovative algorithms that make use of satellite data for the impact assessment. Participants came up with several ideas that are well described in this competition results blog post. The results of the winning ideas were presented to the public through the ‘Rapid Action on COVID-19 with Earth Observation’ dashboard, launched by the ESA and the European Commission. Simply follow this tutorial to explore these results and other interesting indicators.
Visit Euro Data Cube to find out about our subscription plans and take advantage of the free trial options. Additionally, the Network of Resources(NoR) initiated by ESA provides sponsored access to some of our services to qualifying researchers and entrepreneurs. Follow these step-by-step instructions to apply for sponsorship
For more inspiration on the endless capabilities for innovation in EO, the notebooks section of the marketplace has a diverse collection of practical use cases and tutorials. If you run into trouble using any of our services and need support, or have great ideas to share, you are welcome to contact us or post questions on the Euro Data Cube forum. Follow us on Twitter and Linkedin to stay informed on our latest developments.
