This is a short text about how we integrated IATI’s datastore into our platform. Since we started Aidhedge, we hade this idea of connecting our platform to IATI’s database. At first it all felt very straight forward, but as everyone knows, when you start digging you encounter things you didn’t think of.
The purpose of this article is not to provide a detailed technical review, but if you are interested in more details— just get in touch!
We have published about the integration more generally (here), and also about some lessons learned.
1) Getting the data
OK, first of all we need to get the IATI-data in. We have build this around the unique IATI-activity identifier. Using the identifier, we then connect to the IATI API.
Pulling the data from using the API is very easy. We chose to use JSON as a format and download all the activity data with a simple GET request. Working directly with the API ensures that we get the latest version of the data (important as data is published continuously).
The IATI-data is currently published in more than one version. We decided quite early on to support only the latest major version (2.0 and up) of the standard. The versions have quite a few differences, at least when it comes to our specific interests. It just became too much work sorting out the differences (see step 2 below).
In retrospect we should perhaps have looked at projects like OIPA (“OIPA is an IATI data-engine providing a rich and usable API for managing IATI compliant data”). After a quick look at the code, we believe that OIPA treatment and normalization of IATI data can help Aidhedge to support more versions in the future.
2) IATI data => Aidhedge’s format!
Getting the data turned out to be simple. But we pretty soon realized we needed to do a lot of work to convert IATI’s format into a format our Aidhedge engine could work with.
A lot of things are quite simple, getting the activity start date, involved organisations, title etc. But we are mostly interested in analyzing financial flows and transactions, and the big difficulty turned out to be to establish how the money flows.
A major concern for us is to establish:
- Who is reporting the project to IATI?
- Who is the recipient?
- What are roles of involved organisations, and more importantly: who is responsible for the budget?
- In which currencies does the involve organizations work?
- What currencies are involved in transactions?
At times this tuned out to be quite puzzeling and hard to wrap one’s head around! It also resulted in quite a few ”edge cased” and a few too many if-statements 🤦. The main problem is that the data is just not uniform enough!
We have established a flow using Python (programming language) that breaks down the activity data, and sorts through it to map the financial flows. We look for the best data (like transaction currency) and if that’s not reported, we build the analysis on assumptions. Unfortunately it becomes a VERY rough analysis at times.
Example: As most activities dont have data about involved currencies, we establish these from assumptions based on organisations and involved countries. Is the donor DFID? Well then we set the outgoing funds to GBP. The working currencies are often set based on the recipient country. As a majority of activities in reality are implemented in USD, this is a major drawback.
After all this piecing and puzzling the rest was rather straight forward. We had established a simple flow of money in a network that could be analyzed! We save the data to a document in MongoDB and its run thorough all of our own algorithms and analyses, just as one of our “own projects” . And its delivered to the user as report!
Final thoughts
This has all-in-all been a really positive project. We have a first integration live, and have gotten a lot of really useful feedback. We are now considering future upgrades to the IATI-integration.
- Highest on the list is of course to include support for version 1 of the IATI standard. Perhaps this can be solved easily by a move to OIPA.
- Another concern is data quality. We might have to think about the inclusion of some kind of “data quality benchmark algorithm” to give users an understanding of what kind of analysis can be provided. And perhaps give some directions toward better publishing!
- We will also be looking at (or after) more IATI-integrations, to see if there are some functionality we should be including in our platform. There are quite a few that provides nice activity summaries, which could be interesting to include directly into our reports in some way. Have to check who among them that provide open data etc. More interesting would be if we can find some other financial algorithms that are already tailored for IATI!!
- And we are also looking at our own out-put API for financial analysis of IATI-data. Or some kind of widgets, who nows!
