How we learned to care for our customers
I celebrated my 20th anniversary at Microsoft last month. By coincidence, I had been invited to give a talk at the Computing Science School at University of Glasgow. I had graduated from there in 1997 and hadn’t been back since so it seemed like a good opportunity to reflect on and share what I had learned in my 20 years at Microsoft.
Leaving Glasgow
In November 1999 I left Glasgow, Scotland to move to Seattle, USA and join the Visual Studio team at Microsoft. This was literally a dream come true for me. In 1996 I had been lucky enough to get an internship with Intel in Portland, Oregon. Since then, I had been dreaming about getting back to the Pacific Northwest.
After finishing my PhD studies in 1997, I worked for two years for Motorola in Scotland as a developer. I built smartcard developer tools using Visual C++ versions 4 and 5. I never thought that I would ever be able to combine moving to the Pacific Northwest with working on the product that I had been using to earn a living. But on November 7th 1999, my wife and I waved goodbye to our family at Glasgow Airport and left to start a new life in Seattle, and a week later I started a new career as a Usability Engineer on the Visual Studio team.
My first day at Microsoft was on November 15th 1999. Saying I was nervously excited would be an understatement. Excited at the prospect of working on developer tools usability, but nervous because, although I had taken some Human Computer Interaction classes as an undergraduate Computing Science student, I had no formal qualifications or experience in usability.
I joined the team as a Usability Engineer. I was one of a team of five usability engineers supporting a larger team of around 1500 developers, program managers and testers. While the ratio of 5:1500 might seem a little challenging, it’s worth bearing in mind that at that time, many other companies had a ratio of 0 usability engineers to n developers, program managers and testers. Corporate usability was in its infancy (I can vividly recall one interview I had with a company here in Scotland. The interviewer looked at my CV and saw that I had studied HCI while at Glasgow University. He asked what that was and when I said ‘Human Computer Interaction, it’s all about making computers easier to use’ he replied in amazement, saying “We should do more of that” but no, there were no openings for a usability engineer with their company), so getting on a product team with an established usability group was very exciting. Plus, it was an opportunity to work on developer tools, allowing me to combine my two interests — software development and HCI.
Working on the Universal Runtime
When I joined the Visual Studio team in 1999 the team had already been working on a project codenamed the Universal Runtime or URT. This was a new software-based runtime environment and associated SDKs, libraries and programming languages that would allow multiple programming languages to use the same frameworks and run on a commonly available runtime. This would eventually become .NET., one of the most widely used developer frameworks (according to the Stack Overflow 2019 study, .NET and .NET Core are the second and third most widely used framework by developers).
The product team had made significant progress on the URT before I had arrived at Microsoft but there were still some major concerns. For instance, developers using Visual Basic 6, a Microsoft language that was incredibly popular at the time, would have to make some significant adjustments to use Visual Basic .NET. For many developers, this wouldn’t be a small undertaking. Along with adapting to changes to the language (such as zero-based numbering), they would also have to adjust to a new framework that replaced a lot of what they were used to (e.g., the FileSystemObject class) with new classes (e.g., System.IO.StreamWriter and System.IO.StreamReader). There was some concern that our Visual Basic customers would find it awkward to transition to the updated language and frameworks.
There had been a lot of internal debate about this with some believing that professional developers will easily adapt but others (mostly people who worked on VB at the time) worried that most VB developers won’t adapt easily and will instead not make the move to .NET.
The Study — Writing Text to a Text File
As a new, inexperienced usability engineer, this was a great opportunity for me to help our product teams learn more about our Visual Basic customers. We set up a study in our usability labs where over the course of four days we would ask eight participants to spend two hours each trying to write some VB .NET code that would write a line of text to a text file. For most developer frameworks, this is a task that should be achievable without significant help within a couple of hours. Even though this would be the first time that participants used VB .NET, it was such a common task that I thought it was still reasonable to expect people to complete this in the time given.
Eight professional VB developers were recruited from the local area and invited into our usability labs. In the labs, participants worked at a computer with Visual Studio .NET installed. They were asked to describe what they were doing and thinking as they attempted to write VB code that would write a line of text to a text file.
Each usability lab is split into two rooms by a one-way mirror. There is an observer room where everyone can see through the one-way mirror to the participant. There are feeds from multiple cameras that are positioned in the participant side so that the participants facial and body expressions can be observed. We can also see what is on the participants screen and hear what they are saying as they work on the task.
On the participant side they can’t see through the one-way mirror but I have a microphone on my side that allows me to speak to the participant and give them instructions.
While it is a slightly unusual setup (most developers aren’t used to having cameras recording their every move) once participants get into the challenge, they tend to focus on completing the tasks rather than worrying about what they look or sound like.
While we thought that this would be an easy task and one that would likely be over in about 30 minutes, we were very wrong. Rather than completing the task easily, not one participant completed the task in 2 hours without huge amounts of help from the UX researchers. Participants had an awful time trying to learn about the new classes that they needed to use to complete this simple task. Even with a full set of documentation and state of the art design time tools such as IntelliSense, participants were not able to complete this task.
The Aftermath
From our observations, we knew that the worries of the VB team were well founded. If what we saw in the usability labs was anything to go by, some of our customers were going to have a difficult time trying to adapt to VB .NET. We needed to convince the .NET team to modify the design of the .NET framework to provide VB developers with a better experience. While many members of the .NET team had been there to observe some of the sessions, many more members hadn’t been present during the sessions.
After the study we compiled lots of metrics such as success rate, time on task, number of errors made etc into a detailed report and presented this to the .NET team. They listened intently to the feedback we provided and while they believed in it and trusted the data, they felt it was too risky to make any changes now. After all, we were close to shipping, perhaps just over a year away (back then, being a year away from shipping was considered ‘close’ to shipping. Changed days now…). Who knows what regressions might be introduced if they start making fundamental changes to the .NET framework at this point?
We understood the reasons for their response, but we weren’t satisfied. Not after what we had observed in the labs. I remember thinking how bad this would be for our customers and our business if we shipped like this. I knew we had done good work and I felt proud to be involved in this. But I knew that I would only ever look back on this episode as a failure if we couldn’t motivate the team to improve the design of the .NET framework for our VB customers. Nor did I want to have to try to explain why it was ok to ship like this to my old colleagues at Motorola when they upgraded to the latest version of Visual Studio.
But along with my own pride, what was really driving me was this feeling of responsibility I had for the participants we had just observed in the lab. They now represented our customers to me. How could I sit back and watch that and not try to do something for them? My colleagues in the usability team were similarly convicted. So we came up with another strategy, another way in which to try to convince the .NET team that changes were required.
Provoking Empathy
Not only did we have lots of metrics about the study, we also had over 16 hours’ worth of video from these sessions. These videos visibly demonstrated the experience that participants had during the studies. We decided to collect a series of video clips that demonstrated the experience that we observed. I remember spending the next few days feverishly looking for highlight clips and cutting and splicing real video tape (remember, this was 2000/2001). After a few days, I had compiled a set of about five or six good representative clips that showed how awful the experience was.
For example, one of the clips I created was from a participant who had been working on the task for over an hour. Imagine the scene — in the clip you have the whole frame showing the editor and the code that the participant is working on. In the corner of the frame you have the feed from a camera that is positioned right above the monitor that the participant is using. So basically, you are looking directly at him as he works. As the clip starts, he is looking at a compiler error in the editor. His shoulders are slumped, he has a heavy look of tiredness and disappointment on his face. He reads out the compiler error. “Cannot convert instance of System.IO.Stream to System.IO.StreamWriter…” he sighs and slumps even further into his chair. “No clue” he calls out in despair. Then, looking up at the ceiling, directly at the speaker from where my voice has been coming from, he pleads “Are you really gonna make me suffer through this?”
The clip, like most of the others, only lasted about 30 seconds. But when the .NET team watched this clip along with the others, their reaction was immediate. Instead of seeing any changes as too risky, they realised it would be far riskier to ignore them. By empathizing with what the participants were going through, they could truly appreciate the pain they would be inflicting on Visual Basic developers. This inspired them to act and start to devise ways they could improve the overall experience.
Turning Empathy Into Action
One idea the product team had was to introduce more code snippets. A lot of the video clips demonstrated how the participants were struggling with the documentation and translating what they were reading into their own code.
It seemed like a relatively easy fix for the product team. If they would add more code examples that covered more of the common scenarios Visual Basic developers might see, it could go a long way in helping those developers move their projects over to .NET. The team was inspired to rethink their strategy and look for solutions that would cost little, but significantly improve the experience for Visual Basic customers. That’s the power of empathy. The team could feel their customer’s pain and they felt compelled to act.
However, spending more time with the video clips and really paying attention to what participants said and what they did, it was clear that adding more code snippets would not be sufficient. From watching the videos, the reason that they spent a lot of time looking for help in the documentation was not because there were too few code snippets, it was because they were searching for parts of the framework that didn’t exist. For example: they might look for FileSystemObject, a class that they had used for years to write text files to the hard drive, but they couldn’t find it in .NET. The way the team had designed the framework, developers were supposed to use StreamWriter; a class they had never heard of and wouldn’t even consider looking for. These sorts of translations were foundationally important to learn if Visual Basic customers were going to be successful with .NET.
Over the next few months, the team had redesigned the experience. Various object classes were redesigned so that they better reflected the way that VB developers expected to work with text files. We ran the same study again with the same participant profile (but different participants — some from the first study didn’t want to come back) and this time the experience was much better. Within 30 minutes participants were able to use these new classes to write code that would write a line of text to a text file.
In the Customer’s Shoes
With empathy comes a motivation to introduce a change, a change for the better. Watching participants in the lab helped the product team focus on how to improve the experience for customers rather than focus on the person who speaks the loudest, who argues the longest, or who has most seniority.
Addressing the loudest, most persistent or most senior person in the room during a design conversation is an easy trap to fall into unfortunately. Even with all the best intentions, it can be hard to avoid.
At Microsoft we practice what we call ‘dogfooding’. It is a practice followed since 1988 when Paul Maritz took inspiration from a dog food commercial that showed the chief exec of a company that made dog food, feeding the company’s dog food to his own dogs. Paul Maritz wanted to increase usage of Microsoft products within Microsoft since if the products weren’t good enough for us, they aren’t good enough for our customers. The practice continues to this day with the main intention to find major bugs before the customer does.
However, there is a side effect to dogfooding. It can pull us into the false belief that we, and we alone, have the best understanding of the problems that exist in our own software. The more we use our own tools and services, the more we fall victim to this fallacy.
The reality is that our customers don’t always experience problems the same way we do. They don’t always use our products the way we use them. They don’t always have the same needs we do.
Lessons Learned
This study taught me two major lessons, lessons that have guided me during my whole career at Microsoft. First is that to understand the experience, it’s vital to understand the user. Looking back, this lesson seems so obvious now but our first reaction when we reported on this study was to compile metrics, charts, graphs and lots of other statistics. We were called Usability Engineers after all. We were focused too much on the task and not on the user experience. It was only after we saw the reactions of the product team to those statistics that we realised that the metrics weren’t motivating the team sufficiently. So rather than hiding the participants behind a wall of statistics, we let the participants speak for themselves through the collection of video clips. These clips provoked a richer understanding of the participants and helped the product team empathize with the participants, such that the product team was motivated to make the necessary changes to improve the user experience.
The second lesson was that successfully delivering a great user experience is as much about who you are designing with as it is about who you are designing for. When I was creating those video clips, I was thinking about how to convince the product team that something needed to be done. I was thinking about how best to engage with the product team, what would resonate with them. It wasn’t enough for me to write a report, throw it over the wall to the product team and then let them respond. I needed to be responsible for managing and influencing that response. I needed to understand my product team, their concerns, their motivations, in order to know how to convince them that we needed to improve the design of the .NET framework.
Neither of these lessons had been obvious to me or anyone else on the usability team before this study. But from that point on, a lot changed. My title changed for example. Instead of a ‘Usability Engineer’ I became a ‘User Experience Researcher’. I went from finding usability bugs in products that were close to shipping, to designing for our customer right from the start. And I continued to learn and apply many more valuable lessons.
