#4 The real cost of measuring social impact for an early-stage startup
Raising funds for Agruppa, a Bogota-based social enterprise, was a constant battle, especially at the beginning with little or no traction. We were trapped in a chicken-and-egg type of problem, where we did not have the capital to start operations, but no one was willing to give us money either without the market validation from an up-and-running business (more on our fundraising journey and lessons learned here).
This is the exact moment where the World Bank came in to mark the story of Agruppa: equipped with a scientific interest in the potential social impact of a service like Agruppa, rather than with the nagging pressure to make successful (read: profitable) investments that most funds approached us with. In other words, the World Bank was to become the perfect partner to fuel the early-stage development and growth of Agruppa in exchange for learning about our potential social impact.
What happened? Someone in an office in Washington D.C. decided that the World Bank needed more “out-of-the-box” interventions for productivity growth of micro, small, and medium-sized enterprises, applicable in any country. So they launched a call for ideas, open to internal and external applicants, with a $100K USD prize dedicated to piloting the winning ideas. As if that was not exciting enough, the prize came with the condition — or rather the added value — that the Bank was to finance a rigorous impact evaluation of the pilot projects in addition to the working capital. This sounded like winning the lottery, and that is exactly how we felt when even before having sold a single potato to a single Mom-and-Pop shop, Agruppa won the ideas competition and secured the financing from the World Bank, as well as an impact evaluation to be carried out by Innovations for Poverty Action (IPA) Colombia.
A quick recap of Agruppa’s theory of change: our goal was to empower Mom-and-Pop shops in low-income neighbourhoods by providing them with fresh produce at lower prices right at their doorstep. Using Agruppa, the shop owners would not only access produce at competitive prices, but also save the time and money otherwise spent travelling to the central market. The time savings would then translate into more rested shop-owners, more time spent with family, or on other productive activities (such as opening their shop earlier). This change in business practices, combined with money savings on produce and transportation, should result in overall higher incomes for Mom-and-Pop shops. In a nutshell, that was the theory (and we will talk about whether or not it materialised in a future blog post. Official study results are forthcoming).
IPA was going to carry out a randomised controlled trial (RCT) in order to compare price and cost indicators of shops that had access to the Agruppa service (treatment shops) with shops that did not (control shops). Operationally, this had important implications for Agruppa — an enterprise which at this point merely consisted of the two founders, one employee and a rented garage. Specifically, it meant the following:
- Dividing up the neighbourhoods that Agruppa was looking to work with into blocks of approximately 35 shops each, and to randomly assign these blocks to the treatment or the control group. In total, we mapped 73 blocks, containing approximately 2.500 stores.
- During the period of the RCT, Agruppa was going to offer its service to the 1.250 shops located in the treatment blocks. This meant leaving the control blocks untouched, until the 12-months surveying period was over in their respective treatment block counterpart.
- The RCT had a phase-in design, where Agruppa would enter and offer its service to one new treatment block per week. This schedule was coordinated with IPA, who would survey all shops prior to being offered the Agruppa service.
The design of the RCT was quite a standard one, however, its application at an early stage of a social enterprise was not. To date, we do not know of any other social enterprise that has carried out such a formal and rigorous impact evaluation during the first few years of existence (in case you do — please shout. We would love to hear of other cases). First of all, the cost of an RCT is beyond the reach of any early stage business, and Agruppa was extremely lucky to have the evaluation sponsored and carried out by such experienced partners. But even more importantly, we learned along the way that there are reasons which make an emerging social enterprise and a rigorous scientific study not exactly a match made in heaven. In our experience at Agruppa, having committed to the RCT carried some serious repercussions on our evolution as a business, which — in our excitement about the lottery win — we had not seen coming:
- Standardization vs. iteration: Measuring the change that an intervention generates requires the intervention to stay the same over the course of the study. You cannot deliver potatoes to a store today, and tomorrow ask store owners to pick them up at your warehouse, while wanting to measure the impact your service has on transport costs. However, as an early-stage business, this is exactly what you want to do — try, test, iterate, and use customer feedback to continuously improve your service. Committing to the RCT meant we could only make certain changes to our service (such as increasing our product portfolio), but not radically pivot its basic shape and form, if we did not want to make the whole study invalid. Like many entrepreneurs starting-up, this limitation did not worry us at the outset as we were convinced to have nailed value proposition to our customers anyway. Only later did we realise that this was not quite the case, and had to weigh making a pivot vs. our commitment to the World Bank and IPA.
- Planned vs. organic growth: Due to the above mentioned phase-in design, we had planned to increase Agruppa’s reach to one new treatment block per week. This rhythm was perfectly in line with our growth projections at the start, and seemed like a healthy, controlled expansion strategy. What we had not factored in, however, is depending on the company’s strategic focus and cash position, we may want to grow faster or slower than that at certain points in time. For Agruppa, adhering to the phase-in schedule ultimately meant expanding faster than we should have; offering a non-polished service to customers and leaving “burnt soil”; going beyond human resources and system capacities in place at the time; and ultimately growing an inefficient operation to burn more cash than it should have (read more on how we grew too fast on blog #1).
- Natural vs. artificial boundaries for a logistics business: The randomisation of the blocks meant that often treatment blocks bordered with control blocks, creating islands of “Agruppa-land”, where shops could use our service. While we tried to define the blocks based on natural boundaries such as main roads or canals, this was not always possible. The result was a labyrinth of blocks and borders that was logistically difficult to navigate and, most importantly, was inefficient. We were sending delivery trucks further than they would have needed to go if we had been able to offer Agruppa in a geographically cohesive area, and thereby created delivery routes that made little logistic sense. Additionally, communicating route boundaries to drivers with low literacy of maps was a challenge, and even more so trying to make them respect those boundaries when the potential for sales commissions led drivers to recruit customers on their own initiative. It stopped us from telling our drivers to “sell everything, to anyone” in order to avoid cross-selling into control blocks. This was culturally incomprehensible to many of our drivers, and to the business acumen per se.
In hindsight, and despite the logical upsides of counting with the support of the World Bank, accepting the lottery prize of this contract and its attached RCT was counterproductive for a business-savvy, agile, and customer focused evolution of Agruppa. Looking back, we believe the hidden costs of carrying out this type of evaluation at such an early stage may have been higher than the financial and reputational wins from working with the World Bank and running the RCT. However, it came at a moment in Agruppa’s trajectory where no one else was willing to take the risk of financing our vision, which is something we will be eternally grateful for. This in itself should be an alert to all the impact investors out there who claim to finance early-stage companies but really require a lot of traction to bet on them. Later on, the support by the World Bank and IPA was key in raising investment from impact-focused funds and individuals, giving our social impact story a level of credibility hard to reach for an early stage enterprise anywhere. May our partners at the Bank and IPA forgive us for using preliminary (!) study results for fundraising purposes… just another clash of the public and the private sector worlds.
We are proud to have at least partly-bridged this clash which defines social entrepreneurship at its core. Agruppa would not have come to where it came without this early leap of faith by the World Bank. Rigorous evaluations such as an RCT can generate truly compelling data, bringing your purpose as a social enterprise to life. Just make sure you know the costs.
(written with Agruppa Co-Founder Carolina Medina)
This is the fourth in a series of posts about the creation, growth and closure of our social enterprise in Colombia. In post #3 we share our fundraising battle (and success). Check post #5 for why a good value proposition is sometimes not good enough.
