A CoPilot Story (Pt 3): Designing Metrics
Continuing the story
A CoPilot Story Table Of Contents
2. The Product Creation Process and Designing Deploying Via A Docker Compose
3. Designing Metrics
5. Designing The Service & Application Catalogue
In the previous post we defined a narrative (sometimes called an epic in agile development) as a large body of work that could combine many user stories. A user story was defined as the smallest unit of work. I also explain our Product Creation Process that provides structure to the post below.
In this post I am going to explore the design process around Metrics.
Defining Narrative
Although identified as being limited scope (due to third party monitoring integrations such as Data Dog and Prometheus being extremely effective) a user is going to need to understand the state of their service through data visualisations.
Key Performance Indicators (KPIs):
- The user should be able to intuitively understand the status of their service and instance by viewing the metrics
- The user should be able to debug their application effectively by viewing the metrics
This narrative should be used by all personas.
Initial Workshop
As described by my colleague Antonas, after examining the problems that the metrics were designed to solve three different implementations of the metrics were highlighted:
- Thumbnail metrics: these would be visible in the service and instance list view (these are explained in more detail in my colleagues post on Taxonomy and Navigation) and would provide a very short time span: mainly signifying to the user that their service and/or instance was alive
- Service and instance metrics: these metrics would offer a much longer timescale and allow a user to drill down into specific metrics for CPU, latency, memory etc. These would be shown in both the service metrics page and instance metrics (as well as the overview page as described in designing alerts).
- A health/not-healthy icon, provided by the container pilot api, would be visible on all services and instances to provide a high level overview (previously explained here)
During this workshop, as a result of user interviews, the use of boxplots was settled on to be the graphical implementation of the data. To explain the visualisations of these metrics this educational component was designed:
First Testing Session
With six testers interacting with the thumbnail metrics, the findings were as follows:
- 4/6 testers were clearly confused by the graphical representations of the data, although it was clear that it signified that something was happening, the ability to draw any meaningful conclusions from the data was severely limited
- 1/6 testers mentioned that they would have desired more information when scaling their services
- 1/6 testers mentioned that they would desire more information on how the data was collected for the metrics
- 1/6 testers would have desired some more information on how to link the metrics to third party tools, such as Jenkins
Clearly from initial testing sessions, although the thumbnail metrics signified to the user that their application was alive, the boxplots were too confusing for users to be able to deduce anything more from the data visualisation. It was realised that the main reason behind this was that the users were unclear on how to interpret the boxplots.
Iterating Design
The first step in improving metrics was to define how the boxplot visualisation could be improved so users could more intuitively understand what the boxplot was representing.
There are a surprising number of different ways to illustrate boxplots, with one important improvement being to include outliers (data points representing data more or less than 1.5 times the interquartile range).
In this iteration it was also decided to simplify the illustration of the boxplots dramatically with only having one way of illustrating the box plots, compared with the six different ways that were in the previous design.
As a result the below educational component was designed that would be displayed at the top of the service list view the first time a user arrived on the page:
Second Testing Round, Section 1
Although we felt we were moving in a very positive direction we needed to test our new designs to increase our confidence.
Below illustrates the new metric designs implemented into the service list:
And the below illustrates the new metric designs on the project overview (see designing notifications (coming soon) for more information on this) page:
These new metric designs relate heavily to the designs accomplished for notifications, which are covered in this subsequent post (coming soon).
These prototypes were presented to users to elicit feedback. Our testing script for this round are here.
Second Testing Round, Section 2
As well as the comprehensive prototype we also designed and conducted an in depth qualitative analysis of different visualisations of metrics: beyond that of boxplots.
Testing subjects were shown a series of twelve slides, each with different graphical representation of data. In each test the slides were randomly ordered.
The slides differed in having three types of visualisations:
- Boxplot:
2. Quartile:
3. Line (with 3 instances):
4. Line (with 9 instances):
Across these three visualisations, three different sets of data were also illustrated:
- A spike in data
- A sudden change in data (a spike that persists)
- A trend in data
These three sets of data were analysed in our initial research. Outliers were initially outlined as a fourth set of data, however this was decided not to be included due to the difficulty users had shown in understanding this concept in previous testing sessions.
All users were presented with all conditions, meaning that this was a within subject test. This method of testing was chosen so we could test with a limited number of users.
For each of these slides users were asked two questions that served as great stimuluses for deducing a users’ understanding of what they were being presented with, these two questions were:
- What can you determine about the health of your service from this graph?
- What their next steps would be after seeing this graph?
You can view the slides for this test here:
Container Pilot Dashboard Metrics Test
Container Pilot Dashboard Metrics Test 23/05/17
docs.google.com
Results From Second Testing Round
The results from the comprehensive prototype were that most users found that the thumbnail metrics and health icon meant that things were clear indicators that the services were alive and running well.
On the other hand, it was mentioned that the outliers were confusing and that the users would have desired a link to actually see their Wordpress blog up and running: it seems metrics are useless unless users can see things in the real world!
The results from the qualitative analysis were more insightful.
Boxplots: although users understood they were looking at boxplot graphical representations, they really did not understand the intricacies of the inter-quartile ranges (although it must be noted that the median was easily understood).
Some interesting comments on boxplots were as follows:
“I will assume that this is a kind of box plot”
“Not sure about light vs dark blue parts”
“Boxplots are fairly noisy and hard to read”
“I would have this for 15+ instances”
Line: every single tester highlighted how easy the line graphs were to understand, however it was highlighted that line graphs would become extremely noisy with a great deal of instances.
Some comments on line graphs were:
“Nice and easy, a piece of cake”
“I like the ideas of line graphs, when compared with boxplots”
“The one with the three lines is much easier to understand when compared to box plots”
Quartile: a strong 70% of users understood this visualisation, with the remaining users understanding it after some explanation for the tester.
Interesting comments were:
“All makes sense to me”
“Looks like a stock market graph, this is an average graph with difference percentiles”
Implementing Improvements
After analysing the feedback from the testing session, it was decided to create an iteration that would take the advantages of both line graphs and quartile visualisations.
The new design iteration implements a new settings button on the service list that allows people to change the method of visualisation and also units for the graphs.
This design also includes some visual components for changing the thumbnail metric that my colleague Antonas has written up here (coming soon).
Next steps
These improvements need to be tested!
A CoPilot Story Table Of Contents
2. The Product Creation Process and Designing Deploying Via A Docker Compose
3. Designing Metrics
