Scaling Up
Late last year I wrote about how more direct engagements with our customers increased the empathy that the Visual Studio engineering teams had for those customers. With increased empathy comes increased advocacy. All from putting yourself in the shoes of your customer and experiencing what they experience.
It seems odd to say it now, but previously, we had never really focused on the customer experience, we had focused on looking for usability bugs. For example, I remember one of the first ever usability studies I worked on, on the new project dialog for C++. During the study, we investigated how easily somebody could create a new C++ project. We identified some bugs and fixed them in the new dialog. But we never asked what people want to do when they get started. We never thought about the most common experience. For example, how often do you create a new project? More often than you continue working on an existing project? Probably not, but back then, we had some UI and we needed to test it.
Shifting from Usability to User Experience
Around the same time that we completed the first API study on the System.IO classes and realized the importance of empathizing with the customer, my title changed from ‘Usability Engineer’ to ‘User Experience Researcher’. It reflected a big change throughout the company.
For us in the developer division, we started doing many many more studies. Especially around APIs. The videos we had created showing the challenges participants had faced in the first System.IO study were compelling and went viral around the company. Lots of API designers around the company saw them and nobody wanted to be responsible for creating new APIs that would have a starring role in the next round of API usability howlers.
We were a pretty small team though, only five of us who could design and run UX studies. We ran into a problem we hadn’t really experienced before. Demand for our services outstripped supply. We needed to find a way to satisfy the demands for getting user feedback on the experiences that teams were designing for developers.
We took two approaches to the problem of scaling up to meet the demand. We developed:
- A set of observation informed guidelines.
- A set of workstyle based personas
Observation Informed Guidelines
After some time running studies with different developers, patterns of behavior started to emerge. Even though many of the studies were on different APIs and had participants performing different tasks, we started to see some common behaviours. For example, for a particular group of developers, whenever they were working with a new API, they would explore how the API works through the API itself, in the editor, rather than reading documentation. They would use intellisense to discover what classes were available and then, after creating an instance of one of these classes, they would use intellisense again to see what methods and properties were available on those classes.
Whenever these developers worked with classes that exposed default constructors, they would do quite well exploring the types available to them. They could create an instance of a class in the editor very easily and then use intellisense to see what the class exposed. If it didn’t expose anything interesting, they would delete their code and investigate another class.
This pattern came up numerous times, so much so that we decided to study it more formally. Our hypothesis was that for a specific type of developer, that developer will be more successful learning how to use an API if classes in that API expose default constructors.
Studying How Developers Instantiate Objects
We brought thirty participants into our lab and asked them to work on a series of tasks, some of which required them to use classes with default constructors and some of which required them to use classes without default constructors, which instead exposed required constructors. Required constructors were the preference within Microsoft because they prevent the user from getting into an inconsistent state. Furthermore, the shape of the class itself documents correct usage of the class. If the class can’t be used without some state being set on the class, then you shouldn’t be able to create instances of that class without setting that state.
Our results confirmed what we had seen in our previous studies. Most participants preferred to work with classes that exposed default constructors. When we asked them why, they offered multiple reasons. Some preferred the ease with which they could explore the objects available to them, making a clear distinction between code that they are writing to explore an API and code that they eventually write to use that API properly, once they have determined to use that API. Still others suggested that it provides more control, that they can better handle errors in setting class state by setting properties of a class individually and dealing with any errors that might occur. We called this pattern ‘Create-Set-Call’ and wrote it up in a paper that was presented at the International Conference on Software Engineering: https://ieeexplore.ieee.org/abstract/document/4222614
This insight helped us scale. By describing the insight as a design guideline for all teams building APIs for the .Net framework, we could increase the impact that our studies were having so that many more APIs could benefit, not just those APIs that we had studied. But not only that — understanding why this pattern emerged and the behavioural styles that were best satisfied by supporting this pattern — in other words, empathising with the customer — helped engineers at Microsoft see how their customers would sometimes think and behave differently to themselves. When looking at the guideline in isolation, without any customer insight to motivate it, it is easy to justify why the guideline should be ignored. It is only with the additional customer insight and empathy that the guideline becomes truly useful.
We continued to take this approach, looking for and studying other patterns of behaviour and encoding them as guidelines once they had been further verified. For example, another pattern we observed was around the use of the factory and builder design patterns, popularized by the Design Patterns book from Gamma and others.
Guidelines are one way to scale but they don’t always offer an answer for every design question. Sometimes there just isn’t a guideline that can help.
Workstyle Based Developer Personas
We knew that it was important for the teams designing the APIs that would go into the .Net framework to put themselves in the shoes of their customers, to empathise with and advocate for their customers, rather than design something that they think they themselves would use. However, we simply didn’t have the resources for every single team to be able to do all the studies on their APIs that they deemed necessary.
But we could create models, or personas of our customers, that described common behaviours, motivations and work styles and ask the teams designing the APIs to use these models to help answer questions about how customers would expect APIs to work. We had seen patterns of behaviors not just in the way that developers used APIs but also in the way that they debugged their code, in the way that they learned about APIs.
These patterns highlighted different approaches that developers take when working on different tasks. We created models for these working styles and named them opportunistic, pragmatic and systematic:
These terms really helped explain the different ways of working but they didn’t really excite the different product teams. We were very careful to make sure that these were seen as work styles, not as stereotypes and that one individual developer might embody multiple work styles depending on the task that they were performing. We asked the API design teams to think about the most typical workstyle that they should design for and created lots of materials for them to learn about these work styles and learn how to apply them to their designs. For example, we created a questionnaire that they could use to ask real developers how they approach various tasks such as learning about a new API, their preferences for the scope and size of classes that they work with etc. The results from these questionnaires and other research could help them determine what work style they should optimise their design for.
Persona Advocacy
Around the same time, the VB team wanted to capitalise on the insights that they had gained and the empathy that they had generated for the VB customer from the first file I/O study. They wanted their customer to come to life for everybody that would design an API for the VB developer and were keen to latch on to the idea of a work style but they wanted to humanise that workstyle. It’s very difficult to empathise with a work style, it’s quite an abstract concept. Instead, they wanted teams to be able to empathise with a human. They wanted something that teams could rally behind and advocate for.
In previous releases of VB, they had some success by adopting a persona, a description of a typical VB customer. This persona described the type of work that they normally do, the type of organisation that they normally work in etc. It gave life to a lot of statistics about customers. The persona hadn’t really been in use for a while, so the VB team took the opportunity to resurrect the persona and so the VB developer, Mort was reborn.
Within the Visual Studio team, we had two other language teams, the C++ team and the C Sharp team. Management of both teams wanted to increase customer empathy for their customers and so they too developed customer personas. Choosing a good name for these personas would prove to be challenging (I remember a professor at Glasgow University telling a whole class of software engineering students that one of the hardest things about software development was choosing good names for the variables, functions and classes that we were creating. At the time it seemed incredible that naming could prove to be so challenging and controversial…)
The C Sharp team happened to have a life size cutout of a young Elvis at the end of a corridor where most of the C Sharp team offices were. They had grown fond of that cutout, so they decided to name their persona Elvis.
Up until this point, the C++ team had been working away in the background. All of the attention in the division had really been on the new C Sharp language and the VB language. The C++ team, just like the VB team, wanted to bring their customer back to life and make them the centre of everybody’s attention. To that end, they decided to create a persona and name their persona Einstein.
Looking Back
At a time when it was still difficult to have regular, direct interactions with customers, the personas were intended to help the different teams rally behind and advocate for their customer.
Internally the personas achieved their goal. The teams were able to rally around their customer and found it a lot easier to ask what would <insert persona name> here do, rather than what would I do? The combination of increasing empathy and a model to understand how the customer thinks was successful, for a while.
Personas are great in the moment, but they require significant time and effort to create them and then to continue to maintain them. Just as in real life, if the personas don’t move with the times, they become dated. And that is what happened with our developer personas. As technology advanced and the way that developers work changed, the personas looked stale. For example, we never updated them significantly once mobile became a huge platform. Teams naturally referred to them less and less. And we ourselves began to question if stereotyping our users in this way was really the best way to design for our customers.
