Digital Experiments Yield Real-World Results
At the CODE@MIT conference, researchers show how complex online experiments can impact everything from public transportation to job hunting.
By Peter Krass
Designing large-scale experiments on digital platforms can be highly technical, but the results often yield practical solutions and fine-tuned data that address social, public policy and business challenges.
Among the topics featured at the recent 2024 Conference on Digital Experimentation at MIT (CODE@MIT) were Generative AI for software developers, the monetary value of personal privacy, and new laid-off employees can find work.
The 11th annual conference, organized by the MIT Initiative on the Digital Economy (IDE), featured large-scale, randomized digital experiments in the fields of computer science, economics, engineering, entrepreneurship, management science, marketing, product design and public affairs.
These new types of studies produce and analyze greater amounts of data at much more in-depth levels than earlier data experiments.
Here are some studies presented at the event:
Global Data
- Susan Athey, a Stanford Graduate School of Business professor specializing in the economics of technology, discussed an experiment that investigated whether laid-off workers could get new, better jobs by displaying their training credentials online.
Athey and colleagues conducted two experiments involving women in Europe. In one, the women took online courses on the Coursera site, and then displayed their credentials on LinkedIn.
The treatment group of disadvantaged workers — some with no college degree, others from a developing country — enjoyed a nearly 6% improvement in finding jobs. By sharing their credentials, the laid-off workers got back into the labor pool. The findings are important, Athey explained, because many job-seekers wonder whether nontraditional credentials have value in the job market. As the experiment found, these credentials do indeed help.
- Another online experiment involving public transport was presented by Daniel Björkegren, Assistant Professor of International and Public Affairs at Columbia University. With his fellow researchers, Björkegren conducted an experiment in Lagos, Nigeria, where nearly three out of four trips are made via an informal network of some 75,000 minibuses.
The city of Lagos decided to improve its official bus system by launching 40 new routes and adding 820 new buses. Björkegren and his colleagues wondered how the riders of those 75,000 minibuses would respond to the city’s plans. To find out, they conducted a wait-time experiment.
The researchers equipped people with smartphones (most Nigerians don’t own a smartphone) and instructed them to offer random cash payments to bus riders willing to wait longer for a city bus. The upshot?
Data from some 640 bus passengers showed two things: One, that people were willing to wait when they were paid for the inconvenience. Second, that they were willing to wait longer when they knew the wait time in advance.
While it may seem obvious that financial incentives pay off, the results point to other, more interesting lessons, Björkegren explained. Private transit does respond when a city launches new bus routes and added buses. Moreover, cities need to manage hybrid transport, a mix of formal and informal systems, he said. And digital apps, with human oversight, can allow cities to experiment.
GenAI for Software Developers
- Can GenAI be used to improve the productivity of software developers? Three related field experiments presented at CODE by Sida Peng, Senior Principal Economist at Microsoft, say yes.
In the experiments, more than 5,100 software developers were given Github Copilot, a GenAI code-completion tool, to do production work in their regular work environment. The developers worked at three companies: Microsoft, Accenture, and an S&P 500 company that requested anonymity. Each developer was given a specific task, and the researchers measured whether they could do the work faster or more efficiently with Copilot.
The results were impressive. The pooled results from all three experiments showed a roughly 26% improvement in “pull requests” (a Github feature that lets developers request that a change be added), and a 38% improvement in software builds.
Design Matters
- Other CODE speakers looked at their own research and discussed ways to improve digital experiments. A major topic was the design of experiments how experimental results are displayed.
One big issue is designing simpler experiments that can be used by product managers and other nontechnical people. A practitioners panel led at by David Holtz, Assistant Professor of Entrepreneurship and Innovation at the University of California’s Haas Business School, focused on design issues. The panel featured designers from Netflix; Microsoft; Eppo, an experiment and feature-management platform; and software-launch platform LaunchDarkly.
While panelists disagreed on certain details of execution, all said that designing digital experiments involves tradeoffs.
For example, a designer might want to offer a highly usable scheme, but also needs to display extremely complex information. Or, an interface may need to be very interactive, but at the same time completely intuitive.
Joni Rustulka, Director of Product Design at LaunchDarkly, proposed a solution known as “progressive disclosure.” Essentially, it involves presenting information to a user only when they request it, rather than overloading them with all the data up front. “Novice experimenters don’t understand complex data,” Rustulka said. “So we put the data behind a tab. If you want it, you can get it.”
- Design also comes into play when the results of a digital experiment are presented to a nontechnical audience. Jessica Hullman, the Ginni Rometty Professor of Computer Science at Northwestern University, said that the data is aggregated, encoded and visualized is important because data is increasingly used for critical business decisions. For example, a city agency might use the day’s winter weather forecast to determine whether the roads will need plowing or salting.
In other words, how data is presented and designed should be based on how the data will be used, Hullman said. While that may sound obvious, it’s advice that’s not often followed.
The (Low) Price of Privacy
- Another experiment described at CODE explored the privacy paradox: While people say they value their privacy, they also enthusiastically use online services that compromise their privacy, mainly by collecting their behavioral data.
Tesary Lin, an Assistant Professor of Marketing at Boston University’s Questrom Business School, found that many people will compromise their privacy for as little as $10.
The experimenters offered online users cash payments of up to $50 if they would share personal demographic information. Most were quite willing. The average subject provided this information for just $10. To achieve 97% compliance, the payments rose to no more than $30. Overall, higher-income people tended to hold out for the higher incentives, Lin said. By contrast, poorer, younger and less educated people allowed their privacy to be violated for less.
In sum, CODE gave the Ivy Tower and Main Street a new place to meet — and to compare notes on cutting-edge digital experimentation.
Do more:
· Check out the 2024 CODE@MIT agenda and speaker bios
· Watch all of the 2024 CODE@MIT sessions
· Read more on Medium here.
Peter Krass is a contributing writer and editor with the MIT IDE.