Ten Million Users and Ten Years Later: Python Tutor’s Design Guidelines for Building Scalable and Sustainable Research Software in Academia

This post, written by Philip Guo, is adapted from a set of TikTok videos he created to summarize his UIST 2021 paper: Ten Million Users and Ten Years Later: Python Tutor’s Design Guidelines for Building Scalable and Sustainable Research Software in Academia. This work won a Best Paper Award at UIST 2021.

Grad students across many technical fields spend a ton of time writing code for their research. As a faculty advisor, one of the most common questions I get from students is: “How much time should I spend refining my research software and getting users versus just publishing a paper and moving on?” I want to give one extreme answer to that question.

Twelve years ago, back when I was a grad student, I created a piece of research software called Python Tutor, which is an educational visual debugger to help people learn to code. But instead of just publishing a paper and moving on, I’ve been refining the Python Tutor software over the past decade to the point where it’s now reached over ten million users in over 180 countries.

So how was I able to keep Python Tutor’s software development going over all these years as I progressed through my early career in academia? This paper presents ten design guidelines that I’ve developed to help me do so, which fall into three main categories: 1) user experience, 2) software architecture, 3) software development workflow.

User Experience

Here are four user experience design guidelines that have helped me grow Python Tutor’s userbase over the past decade as a research project housed entirely within an academic lab.

• Walk-up-and-use — if you want to get lots of users, they must be able to just walk up to your system and start using it. For Python Tutor, anyone can go on the website, start writing code, and visualize it right away. No installation, no user accounts, and no logins.
• Should ‘Just Work’ — your system should ‘just work’ no matter what users enter into it, or else they’ll walk away if their first impression is bad. For Python Tutor, I’ve put in the sustained engineering effort to ensure that no matter what sorts of complex code users enter into it, it will visualize something reasonable (or give a targeted error message).
• Sharing, Not Hosting — avoid hosting any user-generated content, since that comes with the burden of storing and moderating that content. Instead, Python Tutor makes it easy to share URLs of visualizations across the web and embed them on other websites so that it does not need to do any content hosting, which helps it scale better.
• Minimize User Options — keep the user experience as simple as possible by having few to no options. Python Tutor currently has three sets of user options, but I still think that’s too many and am trying to figure out ways to cut down even more.

Software Architecture

Next, here are three software architecture design guidelines to help your research software scale and sustain over time, especially when developed within a resource-constrained academic lab setting.

• Be Stateless — if you can structure your app to be stateless, that will help it become more robust, scalable, and secure. Python Tutor keeps no persistent user state, so it easily and cheaply scales across servers.
• Use Old Technologies — keeping up with the latest software libraries, frameworks, and toolkits is a never-ending battle. That’s why for Python Tutor I use super-old technologies that have stood the test of time over the past decade or more.
• Minimize Dependencies — on a related note, maintaining software dependencies can also be a huge pain, as this xkcd comic shows. That’s why I strongly suggest minimizing dependencies on external code. And if you really need to include dependencies, bundle them directly into your codebase instead of relying on external package managers.

Development Workflow

This final set of design guidelines is about workflows for sustainably developing software within academia. Unlike in industry, within academic labs there’s often no long-term funding to pay for full-time software development staff, so you’ll have to improvise.

• Single Developer — since it’s so hard to afford long-term software development staff in academia, I’ve found that the best way to keep Python Tutor going for over a decade is to have it be a single-developer project where I’m the only one who works on the code.
• Start Specific — avoid feature creep by hyperfocusing on one super-specific use case. For Python Tutor, the only thing I wanted it to do was to emulate what a programming instructor might draw on the board when illustrating code execution.
• Ignore Most Users — finally, contrary to the principles of user-centered design, I found that the way to sustain software development over many years is actually NOT to listen to most user feedback, since that often leads to feature creep and a loss of focus.

Parting Thoughts

That was a super-quick tour of my UIST 2021 paper, which is about this broader topic of maintaining research software long-term in academia. Of course, it leaves many open questions, such as:

• What are the tradeoffs of valuing long-term software development over publishing research findings from shorter-term prototypes? I don’t recommend the extreme approach that I’ve taken with Python Tutor for all projects … many of my other projects are more traditional HCI prototypes where the underlying interaction ideas were the main contributions, not the software artifact itself.
• What are the career incentives (or disincentives) within academia for doing this sort of long-term software development work? How have I personally balanced my own career progression while keeping this project running long-term?
• In what other settings can such work be done? Within industry research labs? At tech startups? In a distributed open-source project? What are the tradeoffs of doing HCI systems software development in these other settings?

For more details about these ideas and more, check out the UIST paper: Ten Million Users and Ten Years Later: Python Tutor’s Design Guidelines for Building Scalable and Sustainable Research Software in Academia