3 Things global development needs to know about Big Data and Analytics
An outsider’s perspective on the inability to leverage private sector gains
Joining Palindrome and shifting my attention from big tech and financial services to “global development” has left me with whiplash. I’m really struggling to come to terms with development’s inability to grasp the gains being shared openly by the private sector. It’s an awful waste of groundbreaking research on the one side and good intentions on the other.
Post-Cambridge Analytica, we are naturally all feeling a little nervous about big data and predictive analytics. Some have become so overly-cautious that they indefinitely postpone or withdraw from efforts to catch up with the private sector.
Instead of regressing, how about we use this as a moment to take stock of what responsible data practices already exist and how they can be adapted for global development?
The thing is, you could be doing something equally misguided by just sitting on valuable, untapped insights that have the potential to improve people’s quality of life. With the availability and low-cost of computer processing, mass storage, and open-source machine learning libraries, it is alarming that the development community isn’t doing more (despite lots of hype).
These techniques are disrupting entire industries such as logistics, retail, and medicine. AI is spawning entirely new customer-centric services, tailored to consumers’ every whim and sometimes unconscious desires — think digital personal assistants, shopping recommendations, and matchmaking.
The private sector has personalized AI beer (“Beer 2.0” 🙄), while the development sector is still struggling to use transaction data for real-time monitoring.
Leveraging big data and machine learning may not be easy, but it is accessible and affordable to those willing to think through practical applications. Whilst Funders are increasingly expectant, the beneficiaries and customers deserve better. They’re depending on program managers to get creative and overcome the challenges that major advancements like this bring.
1) The results are in
To the doubters, I say — that ship has sailed. The benefits are overwhelming and well suited to certain challenges facing the development community. Few industries and organizations will be unaffected by the changes underway. The impact and gains are simply too large. The private sector knows this all too well and Gartner estimates that 85% of CIOs will pilot artificial intelligence programs by 2020.
“A.I could be used to make government agencies more efficient, to improve the job satisfaction of public servants, and to increase the quality of services offered. Talent and motivation are wasted doing routine tasks when they could be doing more creative ones.” — How AI Could Help the Public Sector, HBR.
So what are we missing here? Why are we still seeing such a big gap in adoption and appetite? Perhaps our thought leaders, are too disconnected from what’s happening on the ground, or maybe implementers are playing it safe and resist deviating from what they’ve done in the past. My guess is that the incentives are so misaligned that there is limited upside to disruptive innovation and “oversized impact” is just a hyperbolic trope without intentionality.
2) They’re coming for your lunch
The adoption of technology seldom favours the well-intentioned. Sometimes, the ones who are able to move early are those with less to lose, and the results aren’t always quite so savoury.
When banks began derisking and pulled out of disenfranchised minority communities in the US, they left an already vulnerable subpopulation even more exposed. It didn’t take much for predatory lenders to fill the void and prey on these desperate consumers, trapping them in a cycle of debt.
It took years and substantial financial destruction before traditional lenders began adopting alternative data and advanced analytics to better understand individuals in these communities in order to provide better credit options.
There is an unfortunate supply and demand archetype, that when service providers aren’t able to meet customers’ needs, or they retreat, such that it causes market failure, there are grave consequences. In development, those consequences have far-reaching implications and should be of even greater concern. We have a responsibility to be reaching further and taking on more calculated risks to leverage the latest and greatest techniques available to us; the disparity between supply and demand is too great to keep up the status quo.
3) It’s more accessible than you think
Yeah it’s scary, and yes there are all kinds of barriers, but being paralysed by fear and uncertainty may be worse.
The challenges and risks are well-worn, well documented, and the roadmaps are more refined than ever.
Consider the regulations in place to protect consumers while still promoting a vibrant credit market; regulations such as the NCA in South Africa, Consumer Credit Directive in the EU, and FCRA in the USA. Similarly, we’re seeing a lot more data privacy regulations coming to bear and with it, the framework and guardrails to responsibly share, aggregate and analyse data for the benefit of data subjects and society in general.
We also seem to be at this really interesting time when large organizations want everyone to use and succeed with AI.
They are putting billions into free tools and training to make AI attainable for even small organizations. Moreover, there are organizations who can help with planning, operationalizing, and flattening the learning curve.
Wherever this lands with you, I’d love for you to share whether you or your organisation is working through the complexities and making better use of big data and predictive analytics; or whether I am misunderstanding the real challenges causing development to fall further and further behind.
