Changes to the OpenAQ API — and How to Access the Full Archive of Data
Stay up to date on changes on the OpenAQ API, and sign up for updates here.
If you follow us on Twitter, saw our recent discussions via GitHub Issues, or follow conversations on our #dev Slack Channel, you might have seen that a few weeks ago, we started a discussion around the idea of moving the full records of data ingested into the OpenAQ platform to S3 buckets, rather than making them all available through our API.
This has come up because the ever-growing amount of data in our system has reached a point where we’re coming up against technical and financial challenges to maintain API responsiveness and its accompanying large database.
Through these discussions and by looking at what other open data projects are doing to deal with similar inevitable-data-amassing issues, we decided to limit the API to sharing the past 90 days’ worth of data in our system. On the OpenAQ website, this also means that the data download tool — powered by the API —currently reports back data from the last 90 days.
In the past week and half since these changes, the API has run much more smoothly and we are ratcheting down the size of our database, increasing the sustainability of OpenAQ!
But don’t worry, you can still access the full archive of data! Here’s how:
- Go here for access to the data as it’s added to our system: https://openaq-fetches.s3.amazonaws.com/index.html. This data is updated every 10 minutes when a new fetch runs. This source also contains all the information for each measurement.
- Go here for access to a daily archive of the data: https://openaq-data.s3.amazonaws.com/index.html. This is an older data format and while it contains most of the information you’d be looking for, it does not contain all the fields for each measurement.
We are working to provide additional, more accessible-friendly ways to access the full archive, including adding in mechanisms that enable the download modal tool to access these data from S3. If you have ideas, share them with us (via: firstname.lastname@example.org, a GitHub Issue, or discussion on our Slack channel)!