GNOME Usability Testing Report
Introduction
This report is an analysis of the usability of three GNOME applications: gedit GNOME text editor, GNOME Calendar and GNOME Files. Those applications were Jim Hall’s suggestions as good applications for the Outreachy Internship, and I picked up those three ones because I use the three of them daily and I personally thought it would be interesting to learn seeing other users testing those applications. The focus of this report is the results I obtained and the methodology I used to conduct those tests.
User Tests Participants
To test the applications, I looked for different kinds of users:
- The ones with no familiarity to Linux that have some computer experience with other OS
- The ones with some familiarity to Linux that never used GNOME’s environment or programs
- The ones who already used GNOME’s environment or programs
This way, it would be possible to observe how close a new user is to understand the interface as an experienced user. Some of other data was collected from them in order to keep some information that may be useful at any time, such as their ages, gender and computer usage.
Participant Data Summary
Age
Gender
Computer usage
GNOME usage
Scenario Tasks
The scenarios were written inspired by the suggestions found at Open Source Software & Usability blog. The scenarios here presented were translated to Portuguese at the tests moment, as the testers are all Portuguese native speakers.
For the usability test, the testers got access to a machine running Ubuntu 16.04 LTS operating system with the GNOME desktop, with the installation as it came — no new programs were installed and no settings were changed.
First of all, the testers were told that the main objective of those tests was not to test them or their computer skills, but to test how the software works for them, whether the software is intuitive or not, whether the software is capable of meeting their expectations on what the software is for or not and so on. It was explained to them too that it would be O.K. if they couldn’t finish some task, because it would be useful to know if a task is not that simple to be executed by the tester and why so.
All the tests begun with a brief explanation of what the software is and what it is for. Depending on the kind of tester, I gave them other programs names with similar proposals so that they could compare their usability.
After the volunteers finished each of the scenarios, I applied a questionnaire with some questions about their experience with the respective program, so that I could know their opinion better. I used the Likert scale to model the responses they could choose, where 1 is the closest to a very good performance and 5 is the furthest, and those questionnaires were based on the After-Scenario Questionnaire mentioned at the Standardized Usability Questionnaires book, page 213, 2nd Edition by James R Lewis and Jeff Sauro. The questions were elaborated based on the tasks to be evaluated.
gedit (GNOME text editor)
Preparation
- Gedit was already started for the user with an empty document.
Execution
1.Type a quick note for yourself, describing something that you want to remember later.
Please, start typing the following short paragraphs into the text editor:
Note:
Teresa and I decided to throw a surprise party to João, on his birthday. We will join at 5:30PM on Friday after work, at João’s. Teresa arranged for João’s rommate to keep him away until 7:00PM.
We need to get the decorations up and music ready to go by 6:30, when people will start to arrive. We asked everyone to be there no later than 6:45PM.
Save this note as party reminder.txt in the Documents folder.
2. After typing the note, you realize that you got a few details wrong. Please make these edits:
- In the first paragraph, change “Friday” to “Thursday”.
- Change “17:30” to “17:00”.
- Move the entire sentece “Teresa arranged for João’s rommate to keep him away until 7:00PM.” to the end of the second paragraph.
When you are done, please save the file. You can use the same filename.
3. Actually, João prefers to go by George, and Teresa prefers Júlia. Please, replace every occurence of João with George, and all instances of Teresa with Júlia.
When you are done, please save the file. You can use the same filename.
4. You’d like to make a copy of the note, using a different name that you can find more easily later. Please, save a copy of this note as “George surprise party.txt”.
5. You decide the font in the editor is difficult to read, and you would prefer to use a different font. Please change the font to DejaVu Sans Mono, 12 point.
6. You decide the black-on-white text is too bright for your eyes, and you would prefer to use different colors. Please change the color to the Oblivion color scheme.
After-scenario questionnaire
A. I found it easy to create and save a text file.
- 1. Strongly agree
- 2. Agree
- 3. Neutral
- 4. Disagree
- 5. Strongly disagree
B. I found it simple to edit an existing text file.
- 1. Strongly agree
- 2. Agree
- 3. Neutral
- 4. Disagree
- 5. Strongly disagree
C. Creating a copy of a text file was effortless to me.
- 1. Strongly agree
- 2. Agree
- 3. Neutral
- 4. Disagree
- 5. Strongly disagree
D. I found it simple to customize the program’s presentation.
- 1. Strongly agree
- 2. Agree
- 3. Neutral
- 4. Disagree
- 5. Strongly disagree
GNOME Calendar
Preparation
- The program was already started with the default window.
- Two events were created before the test started: an event named “Project Start” on December 4th, 2018 and an event named “Icecream with Pedro” on January 20th, 2018
Execution
1. You are setting up your stuff to start working on your new computer. You need to create a new calendar only to submit your work events. Create a new calendar to use only on the machine you are: name it “Work” and choose yellow to its color.
2. Now that you have your work calendar, you have to create an event to remember of an important meeting you have with a client. Create a work event named “Meeting with client” that will happen on November 5th, 2018.
3. After you saved the new event, you realised you forgot to set the time, the place and add a note for the event. Find the event you created and add the following changes:
- Set the time from 10AM to 11AM
- Set the place as São Paulo
- Write a note saying “Meeting about bussiness with João”
4. Someone called you asking you what you have on December 4th. Go on your calendar and check out if you have something.
5. You have an appointment with your friend but you don’t remember when it is. Look for an event called “Icecream with Pedro” to find out when it is going to happen.
6. There are some important events that are not on your local calendar. Connect your calendar with a Google account you already have (account information will be provided so that the user won’t need to use its own).
After-scenario questionnaire
A. I found it easy to create a new calendar.
- 1. Strongly agree
- 2. Agree
- 3. Neutral
- 4. Disagree
- 5. Strongly disagree
B. I found it simple to create an event.
- 1. Strongly agree
- 2. Agree
- 3. Neutral
- 4. Disagree
- 5. Strongly disagree
C. Setting a time, a place and writing a note of some event was effortless to me.
- 1. Strongly agree
- 2. Agree
- 3. Neutral
- 4. Disagree
- 5. Strongly disagree
D. I found it simple to search for an specific date in the calendar.
- 1. Strongly agree
- 2. Agree
- 3. Neutral
- 4. Disagree
- 5. Strongly disagree
E. It was easy for me to search for an specific event in the calendar.
- 1. Strongly agree
- 2. Agree
- 3. Neutral
- 4. Disagree
- 5. Strongly disagree
F. I found it easy to connect the calendar with a Google account.
- 1. Strongly agree
- 2. Agree
- 3. Neutral
- 4. Disagree
- 5. Strongly disagree
GNOME Files
Preparation
- Files was already opened for the user with the default page showing up.
- I created a file called “The Hobbit” and put it into “Images” folder.
Execution
1. Yesterday, you re-organized your files and you don’t remember where you saved the copy of one of the articles you were working on. Please search for a file named “The Hobbit”.
2. You don’t think a text file should be placed at “Images” folder. Please, change its place to the “Documents” folder.
3. To help you to find your “The Hobbit” article, you find it better to create a folder called “Movies” to put it on. Do so and move your article to your new folder, inside the “Documents” folder.
4. Files and folders are usually displayed as icons, but you can display them in other ways too. Change how the file manager displays the files and folders, to show them as a list.
5. You don’t have your glasses with you, so you aren’t able to read the names of the files and folders very well. Please make the text bigger, so it is easier to read.
After-scenario questionnaire
A. I found it easy to search for files.
- 1. Strongly agree
- 2. Agree
- 3. Neutral
- 4. Disagree
- 5. Strongly disagree
B. I found it simple to move files through folders.
- 1. Strongly agree
- 2. Agree
- 3. Neutral
- 4. Disagree
- 5. Strongly disagree
C. Creating new folders was effortless to me.
- 1. Strongly agree
- 2. Agree
- 3. Neutral
- 4. Disagree
- 5. Strongly disagree
D. I found it simple to change the icons exhibition.
- 1. Strongly agree
- 2. Agree
- 3. Neutral
- 4. Disagree
- 5. Strongly disagree
E. It was easy for me to change the text size.
- 1. Strongly agree
- 2. Agree
- 3. Neutral
- 4. Disagree
- 5. Strongly disagree
Usability Testing Results
To summarize the usability tests results, I used a heat map as taught on Linux Voice’s November 2016 edition (page 37). In this same file, I could find some other references about how to report the usability testing results. The results of the after-scenarios questionnaires were used to compare with the testers performances. A heatmap was elaborated now taking in concern their opinion.
The colour on the conductor’s evaluation results heatmap indicates the relative difficulty of each task for each tester:
- Green if the tester easily completed the task.
- Yellow if the tester experienced some (but not too much) difficulty in the task.
- Orange if the tester had some trouble in the task.
- Red if the tester experienced severe difficulty in completing the task.
- Black if the tester was unable to figure out how to complete the task, and gave up.
The colour on the testers opinions heatmap indicates the difficulty of each task on their opinion:
- Green if the tester found it very easy.
- Yellow if the tester found it moderately easy.
- Orange if the tester didn’t find it easy nor difficult.
- Red if the tester found it moderately difficult.
- Black if the tester found it very difficult.
It is important to emphasize that not necessarily both of the heatmaps are directly related. The opinion of the tester may diverge on its performance, but it is interesting to see their similarity.
gedit (GNOME text editor)
What worked well?
Users in general did not have issues while creating, editing and copying a text file. I noticed that only one of them had the idea of using the find and replace functionality when I asked them to replace names in the paragraph, the rest of them changed the names one by one, but I couldn’t evaluate this because even I didn’t realise this task could be easier than I thought. On future tests, I should add this task.
What were the challenges?
When they had to change gedit’s font and theme, they were not able to find the program’s preferences.
At the end of the scenarios, I showed them how they could change the program’s presentation and even the ones who already are familiar to GNOME seemed surprise to find out that the option to change it stands actually outside the program, in a certain way.
Some improvement suggestions
The bigger problem is the way GNOME treats some applications functionalities, not actually a gedit problem. Even though it is great that they are not placed where they do not take a big space, they are not very obvious. Maybe if those kind of functionalities were separated from the status bar and in another color they would call the attention better then now.
If the user could change the font size using the ctrl + or ctrl - shortcuts, it would be easier and faster then looking for a Preferences window, though.
GNOME Calendar
What worked well?
It was very intuitive for the users to create an event — being able to create an event only by clicking on a specific day is nice, and they can also do it by clicking on the add event option placed on the top right corner. Most of them found it easy to add other details to the event. Connecting with an online account went well for most of them too, but I believe it was so because they saw that option in the moment they were creating a local calendar.
What were the challenges?
I noticed most of them were a little bit confused about understanding the concept of adding a new calendar to the present calendar, it was harder then doing the task itself. When they were trying to create a new calendar, they couldn’t find the option very easily, since it did not seem intuitive to them creating a new calendar from the option Calendar settings. They passed through this option once or twice before they decided to try to click there, it clearly was not their first option.
Even though it was nice that they could add a new event only by clicking on the day the event should happen, some of them didn’t understand why the program does not show all the event’s fields at once instead of showing a button with an Edit details option.
When they were searching for an specific date, I saw all of them first looking for the option to swipe between months on the right of the application’s screen (what, I believe, happened because they were looking for a month after the actual month, if they were looking for a month before the actual month, they’d probably look for the option on the left of the application’s screen). Some of them tried to scroll using the mouse, but it didn’t respond as they expected.
When I asked them to add an online account, I observed they frequently were confused by the option Synchronize, they thought it could be the option they were looking for, but most of them didn’t click there because they saw the option before, on the first task.
Some improvement suggestions
It’s comprehensible that the concept of adding new calendars to a calendar is a kind of default, as other applications use that too. I think their confusion was caused because they do not use this feature. Maybe if the option Calendar settings was named only as Calendars… they would think the option give them a list of calendars — what actually happens.
As I saw they thinking a while before hitting Add button, I imagine maybe a + button would make it clearer.
When they create a new calendar, I saw that they were confused because the calendar settings does not close automatically. I think it would be a great idea to give the user a feedback that says the calendar was created and close the calendar settings immediately.
Putting the buttons to change the month on the right and on the left side of the screen would be nice, as their first thought when they try to navigate through months is that they are turning pages.
The option to synchronize could be a refresh icon with a cloud on the application’s bar, the user would easily understand it is a synchronize button.
GNOME Files
What worked well?
This program had the best results. Users could find, move files, create folders and change icons exhibition and font size very easily.
What were the challenges?
When I asked them to look for a file, I noticed they didn’t find that easy to tell where the file was. That’s because when the user looks for an specific file using the search tool, the program shows only the file as a result, it does not show the file’s path.
Some improvement suggestions
I would be interesting to show the path where the file searched is, as this information is relevant when the user is looking for some specific file.
