Short Messages Fall Short for Micro-Entrepreneurs: Experimental Evidence from Kenya

Muhammad Zia Mehmood is a PhD candidate in the Business and Public Policy Program at the Haas School of Business, UC Berkeley. In his job market paper, Zia studies the demand for and potential of business trainings provided over text messages to impact outcomes for micro-entrepreneurs in Kenya. This study was supported by CEGA’s Development Economics Challenge initiative. This blog post was originally published on the Econ That Really Matters blog.

A small business welder at work. Credit: Adobe Stock

Introduction

Small businesses form the economic backbone of low-income countries and strengthening these enterprises is fundamental to alleviating poverty. Poor management practices is a major factor constraining firm productivity in these contexts, and over $1 billion is spent annually to address this constraint by providing business trainings to entrepreneurs. However, most of these are conventional, in-person, classroom-style trainings, which are expensive and hard to scale, and can exclude those who are unable to participate in person. Due to their low costs, scalability, and reach, phone-based trainings are gaining popularity as a potential solution, but there is limited evidence on whether remotely provided trainings are effective for micro-entrepreneurs in low-income settings.

In my job market paper, I study the demand for and potential of text message-based business trainings using a field experiment in Kenya, in which access to an SMS-based training was randomized across 4,700 micro-entrepreneurs. I estimate short- and longer-run impacts using phone-based surveys conducted three months (Midline: 307 observations) and twelve months (Endline: 2,780 observations) after the intervention. I also elicit ex ante predictions for 12-month treatment effects from researchers through the Social Science Predictions Platform (SSPP), to assess whether the main findings depart from existing priors. Finally, I measure demand for the trainings through Take-It-Or-Leave-It (TIOLI) offers and the Becker-DeGroot-Marschak (BDM) willingness-to-pay elicitation method for a subset of the sample.

Context and Intervention

According to a 2016 nationwide survey of small businesses in Kenya, 90 percent of micro-entrepreneurs had never received any type of business training. This was reflected in their business practices: More than three-fourths did not advertise any of their products, over two-thirds didn’t keep any business records, and less than a tenth accounted for prices set by their competitors when choosing their own prices.

I partnered with a local firm specializing in digital content development and dissemination to implement an SMS-based training aimed at addressing these management gaps and others. Available in English and Swahili, the training modules covered best practices, including marketing, advertising, pricing, record-keeping, and stock management. The content was structured around stories about the decisions of hypothetical micro-entrepreneurs in different scenarios. Users accessed the trainings through self-paced engagement with an interactive chat-bot, which sent bite-sized chunks spanning about 150 text messages. The entire training could be completed in five to seven hours, and all content was retained indefinitely on users’ phones. Weekly text reminders were sent to those who stopped engaging, and these reminders stopped if engagement was resumed or after two consecutive months of inactivity.

The primary sample for the study was sourced from a list of micro-entrepreneurs compiled by my implementation partner and a local microfinance institution. Half of the study sample consists of female micro-entrepreneurs, and roughly 45 percent is based in rural areas. The average individual was about 35 years old, and had completed almost twelve years of education (high school level).

Figure 1: Screenshots of user engagement with the chatbot. Note: This figure shows screenshots of interactions with the SMS-based chatbot as it pushes out content to users. In this context, most micro-entrepreneurs set prices just based on their buying costs, without accounting for prices of their competitors, so the content pushes them to change their pricing strategy. Credit: Muhammad Zia Mehmood

Results

Three months after the intervention, I find that the SMS training increased knowledge and adoption of best practices by 0.20 and 0.33 standard deviations, respectively. I also find large positive, but statistically insignificant, effects on business performance in the overall sample, and significant positive effects for younger (below-median) micro-entrepreneurs on sales (109 percent increase), profits (38 percent increase), and business survival (11.6 percentage points increase). These positive effects for younger entrepreneurs are driven by higher engagement with the content, and larger effects on time spent on business, and loan amounts applied for and received.

However, these positive results dissipate in the longer run; twelve months after the intervention, I see no effects on knowledge and adoption of best practices, as well as business sales, profits and survival. Additionally, the positive effects on business outcomes observed for younger entrepreneurs at three months disappear after twelve months. The lack of long-term impact was likely driven by micro-entrepreneurs abandoning all interactions with the content within the first few months of the intervention. The survival curve in Figure 2 shows how all cumulative aggregate engagement with the platform ended by May 2022 — five months into the intervention.

Figure 2: Survival curve of interactions with chatbot. Note: This figure illustrates how interactions with the chatbot were distributed throughout the study period. The plot shows reverse cumulative engagement over time; for example, it shows that 80% of all the interactions with the chat-bot throughout the course of the study, had ended by 4/1/2022. The shaded areas represent the time-spans during which the Midline and Endline surveys were conducted. Credit: Muhammad Zia Mehmood

Figure 3: Predictions vs observed treatment effects. Note: This figure shows how predicted treatment effects for the Endline compare with observed Midline and Endline effects. Error bars represent 90% confidence intervals. Credit: Muhammad Zia Mehmood

Figure 3 illustrates how these results compare with predictions for 12-month treatment effects elicited ex ante through the SSPP. I find that SSPP researchers overestimated the engagement levels, both in terms of the proportion of the treatment group that would start engaging with the content (50 percent vs 30 percent) and how much training content the average user would complete after twelve months (40 percent vs 7 percent). Furthermore, predictions for the 12-month treatment effects on knowledge and adoption of best practices are somewhat similar to observed effects at three months, but significantly overestimated in light of observed 12-month treatment effects. Effects on business performance offer a similar story: SSPP predictions for the 12-month treatment effects on sales and profits are similar in magnitude to effects observed at three months (albeit statistically insignificant), but they grossly overestimate the effects at twelve months.

Additionally, notwithstanding the low engagement and lack of longer-run effects, I find positive demand for SMS-based trainings among micro-entrepreneurs; both methods of elicitation — the TIOLI offers and the BDM exercise — reveal that individuals are willing to pay a small amount for an additional SMS-based training, suggesting that they value access to the content.

Policy Implications

These results indicate that SMS-based trainings are unlikely to improve outcomes for micro-entrepreneurs in the long run, despite their growing popularity in low-income and less accessible settings. These findings also highlight the lack of engagement with trainings as a major challenge that limits the potential of remotely-provided information-based support.

Further, the forecasting exercise reveals that social science researchers overestimate the potential of SMS-based trainings to improve outcomes for micro-entrepreneurs, and the findings from this study are thus contrary to priors. Updating these priors is important because policymakers and practitioners often rely on social science experts to make decisions about how to invest in remote, information-based support programs.

Lastly, the results on willingness to pay suggest that engagement with the content might not reflect the actual demand for SMS-based trainings, pointing towards possible behavioral drivers constraining engagement. To capitalize on the full potential of digital content delivery in low-income settings, further research is needed to shed light on how to encourage engagement with remotely provided content.