Getting Data on Demand with EDC Browser
The ability to share and access the cutting edge of Earth Observation algorithms is now easier than before
The Remote Sensing community has generated a huge amount of knowledge and expertise over several decades of research. However, projects have generally been conducted in small test areas and then only shared within small communities and through journals. If you’re lucky, researchers decided to share their code via online repositories, but you have to search thoroughly to find it, and then also put considerable effort to understand and properly use it.
To help tackle this issue, we have developed the Bring Your Own Algorithm (BYOA) service, making it easier for developers to share complex algorithms with remote sensing users around the globe. Users can then access these algorithms through the Data on Demand service. Fully automated processing means that after selecting their desired parameters, such as the area of interest or time range, users have no additional work to do. All data processing is in the cloud and with just a few clicks, the results are simply delivered to the user.
BYOA was publicly introduced at ESA Phi Week 2021 held in Frascati, and we are now tremendously excited to share this feature of Euro Data Cube with you in a series of blog posts as well! In this blog, we will introduce you to the concept and the idea behind Data on Demand and showcase how you can access it through EDC Browser. As EDC Browser will be the star of this blog, we will first briefly introduce the tool, as well as look into a huge variety of data it offers. Lastly, we will also look into each of the currently available BYOA algorithms.
Introducing EDC Browser
EDC Browser is the easiest and best way to access and visualise the numerous datasets that are available through the EDC Public Collections repository, containing, among others, satellite imagery, land cover maps and land parcel datasets. In EDC Browser you can explore no less than 74 (and growing) public collections, including Sentinel, Landsat, Copernicus data, DEM, miscellaneous collections such as Motorways, population density, land cover maps and many more.
You can access EDC Browser with a direct link and explore public EDC Data freely. To access BYOA and commercial data in EDC Browser, you will need to have an account and be logged in. If you have an EDC account, you can deploy your own EDC Browser (to do so, follow this page in the marketplace and click on Deploy app. You can then access the app through a direct link in the left menu of your Dashboard).
On the top of EDC Browser, in the EDC Data tab, you will notice two options — Public and Commercial (if logged in, you will also see a third, User tab, where your personal collections can be viewed). Public tab includes all the public collections from the public collections repository mentioned above.
Using the Commercial tab, you can directly order commercial Airbus SPOT, Airbus Pleiades, PlanetScope or Maxar WorldView data within the application. If you need very high resolution data, feel free to try it out. To order commercial data, you will need to purchase some quota for each commercial collection (if you’re conducting research or precommercial exploitation, you can get commercial data sponsored). When you have your quota, you can order data directly in EDC Browser. After selecting your desired parameters and searching for available data, you will prepare your order, confirm it and wait for automatic ingestion. Tiles will appear in a collection of your choice in your Sentinel Hub Dashboard, and will be available to you in the EDC Browser User tab. See our webinar on commercial data with Sentinel Hub to learn more about the functionality. Although the webinar focuses on ordering data through EO Browser; don’t worry, the steps of ordering in EDC Browser and creating a configuration are very similar.
EDC Browser doesn’t just offer a huge variety of data — it also has powerful visualisation options! Along with visualisations that we prepared for you in advance, you can also use the composite option to create your own RGB composites, the index tool to create custom indices, or use the powerful custom script option, which allows you to manipulate data visualisations using Javascript.
You are invited to participate with adding more data collections to EDC. The greater the diversity of data, the more beneficial the collection will be to the rest of the community! You can find out all you need to know on how to do this in our dedicated EDC Public Collections Github Repository.
Now let’s dive into data on demand!
Data on Demand
A One Stop Destination for EO Products
The vision of BYOA is to allow algorithm developers/providers to onboard their algorithms in as frictionless manner as possible and to make their algorithms available for commercial and non-commercial users alike. By adding as many algorithms to the EDC Marketplace as possible, we want to create a centralised space where developers and scientists can develop and share their work. Having such a common space makes it easier to contribute to the community and even makes it possible to sell their algorithms in a one stop destination for EO algorithms and products. The whole process from data download to data processing is done on the cloud, within the EDC. This is simply amazing, as it puts data access and complex data processing at anyone’s fingertips. Imagine a platform, where a global Earth Observation community contributes to develop and share hundreds of algorithms and data collections, accessed by thousands of users across the globe. We believe this is exactly where EDC is headed — isn’t this exciting!
Ordering data on demand through EDC Browser
The currently onboarded algorithms are not free to use; to use data on demand, you will first need to get some credits to your EDC account. To get credits, visit this page and read this documentation.
Once you have some credits available, navigate to EDC Browser and switch to the On demand data tab on top to access the offered algorithms. In the Algorithm dropdown menu, select the algorithm you wish to purchase. You will be prompted to draw an area of interest — you can either draw a rectangle or a custom polygon, or even upload a GeoJSON/KML/KMZ/GPX file for your geometry. Aside from setting the area of interest, each algorithm has its own optional and required parameters. For example, to run the Maximum Chlorophyll Index algorithm, you only need to select the time range that you are interested in, as well as specify the name of the order. If desired, you may also specify several optional bucket settings to have data delivered to your personal bucket instead of the EDC one. The cloudless mosaic algorithm on the other hand requires no additional parameters, while truck detection algorithm requires several.
After clicking Purchase, you will be sent to a confirmation page where you will be required to confirm your input parameters. The system will then automatically estimate the cost of your analysis in credits for you to confirm. Lastly, after confirming everything and agreeing to the terms and conditions, you will be able to confirm and run your algorithm.
When the results are ready, you will receive a notification linking to your BYOA results. Each algorithm is delivered differently; for instance, the MCI will link you to the AWS bucket where the data is stored, alongside a link to xcube viewer, where you will be able to view the results of the algorithm analysis. Truck detection and the Cloudless Mosaic on the other hand will offer you direct files to download right away.
Algorithms Currently Available
Currently there are three algorithms available to purchase via EDC Data on Demand. Over the coming months more algorithms will be added. Stay tuned for more detailed blog posts on specific algorithms in the future!
EOxCloudless Sentinel-2 10m Global Cloudless Mosaic
This derived product developed by EOX was created to provide global 10m cloud free mosaics on an annual basis using Sentinel-2 L2A imagery. The product is available as a 4 band GeoTIFF. See some cool use-cases, including change detection, global vegetation monitoring and time-lapse analysis, here. If you need cloudless Sentinel-2 imagery in full resolution, this is the product for you!
Maximum Chlorophyll Index
This product is used as an indicator for critical eutrophication. It can be applied to any body of water for the identification of potentially critical situations due to excessive algal growth. It was originally developed for the MERIS platform but has been adapted for Sentinel-2 imagery for the implementation of this algorithm. The product contains the MCI itself, plus the input bands of Sentinel-2 as well as Scene Classification and Cloud Mask, allowing for the analysis of potential issues caused by the contamination with undetected clouds.
The product is delivered in .zarr format and offers global coverage with data going back to June 2015.
Truck Detection
This algorithm utilises 10m resolution Sentinel-2 L2A imagery to detect moving trucks and was authored by Henrik Fisser as part of the Euro Data Cube COVID-19 edition of Sentinel Hub’s custom script contest. It takes advantage of the offset between Sentinel-2 bands which causes a rainbow effect over larger moving objects such as trucks and aircraft in RGB images.
The algorithm results are delivered as point vector data in Geopackage format (.gpkg) with the coordinates of the detected trucks, dates of detected trucks and the valid road area as attributes. The algorithm is especially useful for truck traffic monitoring and management, tracking of disruptions of logistic flow and to complement highway usage data. Find detailed information about the algorithm, along with sample data here.
Onboard Your Own Algorithm
If you have a cool algorithm on your hands, you are invited to onboard it to our on demand data offering. It’s a great opportunity to share your work with the community, and you can even sell your algorithm to end users!
If you’re curious how, stay tuned for our next blog post, where we will go into detail on how you can easily onboard your own algorithms to EDC. We’ll cover how to write the Jupyter Notebook and also the process you will need to go through to make your algorithm available for users and customers through the EDC Marketplace. If you are keen to find out more, don’t hesitate to contact us! You may also visit our official BYOA documentation on more information.
Visit Euro Data Cube to find out about our subscription plans and take advantage of the free trial options. Additionally, the Network of Resources (NoR) initiated by ESA provides sponsored access to some of our services to researchers and entrepreneurs. You can follow these step-by-step instructions to apply for sponsorship.
We are looking forward to seeing what you do with EDC data on demand, and of course, we can’t wait to see your onboarded algorithms!
- To find out more on how to onboard your own algorithm visit here.
- To learn how to use onboarded algorithms, check here.
- For more inspiration on the endless capabilities for innovation in EO, the notebooks section of the marketplace has a diverse collection of practical use cases and tutorials.
- If you run into trouble using any of our services and need support, or have great ideas to share, you are welcome to contact us or post questions on the Euro Data Cube forum.
- Follow us on Twitter and LinkedIn to stay informed on our latest developments.
