Developer Experience Metrics

How to measure DX using UX metrics. :)

Hi people, continuing the series of articles on developer experience now we will address the topic of metrics for DX, the central idea of this article is to bring a list of metrics that can be used to measure whether your product is appropriate or not within this principle. In case you do not have a clear idea of what is DX I recommend that you read the first article in the series.

In my view the first step to create DX metrics is to already have full mastery over UX metrics, but why? Because DX is a strand within UX, that is developers who use your product, even if extremely technical users, are still users who will be impacted if your product has a bad UX.

Users > user experience > developers > developer experience

So, going against the idea that knowing about DX means understanding UX as well, let’s list some pillars within the user’s experience universe and how to measure them.

UX pillars

Usability

How easy is your product for your users? The pillar of usability comes to direct us concerning the experience we will draw for our product, or show us points of improvement within the current experience. To measure usability, I recommend the following methods:

• Guerrilla test (go out and ask who you find)
• Lab test (call a group of people to a specific location and test your product with them)
• Remote usability test (Give the user access to the product and measure their interaction, for this I use hotjar, mixpanel, among others)
• Face-to-face and/or remote interview (Assemble your script and select a group of people to interview, here you can collect qualitative and quantitative data, usually adopt a qualitative data approach and then turn it into quantitative).

Accessibility

Is your product or service accessible to your target users in their respective roles? It is important to map all the people who will use your product and your journeys during this use, for this an artifact called journey map is usually elaborated, in it, you can map all the stages of the journey, also being able to map some important points to your context, such as challenges in each step, below I put the framework that I use to create my journey maps.

As you can see, at each step, I’ve already cast what possible metrics I can use to validate that each step is going well or not, for a product manager this greatly facilitates the process mapping business KPIs and technology.

Findability

For those who are Brazilian and like football, reading this word may seem like a term that coach Tite of the Brazilian national team invented lol. Jokes aside, how simple is it for your user to find what they need in your product?

In this pixel the main way to validate efficiency is by measuring the time used by the user to complete a task, you can do this through laboratory tests with screen recording and user, or else remotely using the hotjar or screen record tool.

A cool thing can be to put in the array of user flows a line of “Time to complete a task”, so you can compare time between several different journey proposals.

Credibility

I believe that this is the pilar that most directly has a relationship with DX, because credibility in products is to know if what you promised has been delivered to your customer, and if there is something that developers do not like, it is people/companies that promise things and do not deliver them.

I worked on a large project of a company that used a giant framework for development, they promised to accelerate the work of developers up to 10x, but they had so many bugs that in the end we were delayed by almost 2x, we left the framework aside and only consummated their APIs when necessary, a project delivered on time.

Therefore, one of the best ways to measure credibility is to measure bugs, simple as well, measure beyond bugs that appear within the product, process bugs due to your user’s lack of understanding of how something works are just as bad.

To measure bugs I use Jira’s service desk, I know that sometimes an additional character in a sentence can be understood as a critical bug by your users and this can hinder your metric, so it is important to have mapped business KPIs and technology so that at some point you revisit the tickets and categorize them accordingly to their real importance and accordingly to the KPIs you have.

Is it possible to apply UX metrics in DX?

This is a question That I asked at the beginning of my journey working with products that were intended for developers, because when researching about DX metrics I did not find any framework or list of methods of validation of experience, however, I found dx pixels that helped me define processes to measure the effectiveness of my product.

When analyzing the dx pixels I noticed a natural correlation with UX with some small changes, so I imagined it possible to apply UX metrics, for this I made a correlation of the pixels of each and possible application of the same metrics. Below I list the dx and UX pixels and their correlation.

[DX] Function = [UX] Accessibility

The basis of the developers’ experience, a dev tool is as good as the function it offers to perform an activity. Good interface, marketing, miraculous promises, and bullshit in general, will not be able to hide bad features. If it doesn’t work, it’s no use, there’s no DX.

One thing I noticed in many developers tools is that they solve the challenge they propose, but do not have good accessibility, either because the product is being used by someone who is not the target audience, or because the journey was not designed for the different types of users who will be in contact with the product.

For this point, it is important to make the journey map of all the personas that will be in contact with the product, so you can outline a journey for each thinking of its particularities.

[DX] Stability (Uptime) = [UX] Credibility

In addition to working, your product must have high performance and reliability, of course, the software is subject to bugs, so it is important to quickly troubleshoot errors in the product so as not to cause great damage to users.

Uptime in the relationship of trust with your product begins to be built, with it, the perception of value drops dramatically.

Part of a product’s experience is provided by the degree of credibility that the product passes on to its customers. In the case of products for developers, this is the degree of stability that this product provides since the uptime of a platform in addition to giving headaches to its users can generate losses of millions of dollars.

For example, when AWS or Azure falls for a few moments, it loses all your customers who use this service and consequently, your customers’ customers. For me, stability is one of the strongest piers within DX.

To measure uptime, I recommend using tools that measure uptime as Statuspage.io of Atlassian for ease of implementation, flexibility to measure various services, and a good experience for those who will consume the service.

[DX] Ease of use = [DX] Usability

Ease of use in DX is beyond what it seems, it is not only about navigating the tool, but also about accessing what is needed at all stages of the journey quickly and efficiently.

Rich documentation, use cases, communities, knowledge bases, keyboard shortcuts, snippets, intuitive filters, previously saved searches, also, deeper points like performance together add speed to the developer’s interaction process with your product and increase engagement.

In UX we say that it is common to say that good design does not often need to be explained, but in DX this is a little different because sometimes your product is a horizontal offer (horizontal offer = applicable to various segments and different use cases) and the first step to provide a good DX is to understand what challenges to be solved within by your customer is.

To measure ease of use I recommend using feedback tools within the platform such as Hotjar or Mixpanel (I find hotjar more affordable in terms of costs but less complete as a solution), and the trigger for these feedbacks should appear at the end of a key point of the journey.

[DX] Clarity = [UX] Findability

At this point DX is committed to providing a simple interface that brings the information needed by the developer to do his work, helping them with critical day-to-day actions. Clarity is concerned with providing the developer with full visibility into the possible consequences involved in an action and the history of its actions.

Another common mistake I see on platforms for developers is to present all the functionality they have for a persona at the same time, sometimes even having mapped the journey to that persona, this happens by the fact that being all visible is better for the user when finding something they need.

But this is not correct. The ideal situation is to present all the functionality according to the user’s evolution within the product, according to the specific use case. On very large platforms such as the Atlassian suite, they redesign their navigation to orchestrate the various functionalities between products (common user view and admin) in two menus and a general search.

To measure findability and clarity I recommend it to be used again or Hotjar and Mixpanel for collecting heatmaps on specific pages (in this case only Hotjar has heatmaps), and also for mapping navigation funnel step-by-step and measuring breakpoints of this funnel.

Conclusion

Finally, I believe that UX metrics are valid, to measure DX and the only set of metrics you need to use, as long as you make the necessary adjustments to your context, this is the most important point, because, in the end, you need to deeply understand the people, scenarios, challenges, and objectives that will be in contact with your product, only then can you choose assertively which metrics make sense to measure your business success.

We’re here in this article, in the DX series. In the next article, I’ll cover how developer communities can make your product grow absurdly or die in the blink of an eye.