Usability Evaluation of Ello.co

We made recommendations to this new ad-free social network and the changes are evident.

Here we go…

Goal

Conduct expert review and usability evaluation the Ello.co (an ad-free social network) for both expert and novice users based ten tasks for each user.

Overview

An initial expert review was conducted, and a usability test plan was designed based on the issues identified in the expert review. Two kinds of users were involved: Novice User and Existing User. Both users completed ten tasks each. The tasks were designed from the core features of Ello website such as requesting an invitation, adding a header image, locating notifications, deleting a post, formatting a part of the text and more. Users also completed a pre-test questionnaire and post-test questionnaire, which collected demographic data and assessed the overall usability of Ello (SUS). We also recorded the time on task to complete each task for both new user and existing user.

What is Ello.co?

“Simple, beautiful, and ad-free”. This is Ello, the new social network created by a small group of artists and designers in 2014.

Ello has one belief. Ads are tacky, they insult our intelligence and we are better without them. Ello is different. Unlike other social networks, which are run by advertisers and users are essentially the product that is being bought and sold, Ello doesn’t sell ads nor do they sell data about users to third parties.

Social networks are clever. While being a “free” service, users in return have to pay the high price in intrusive advertising and a lack of privacy. With Ello, a user can opt out of information sharing.

This is how the landing page looked like.

Major findings from Expert Review

We used Jakob Neilsen Heuristic Evaluations to test the usability of the site.

The findings are as follows:

  • Unclear Icon Meaning
  • Lack of Visibility of Toggle Buttons
  • Poor Icon Visibility

Cognitive Walkthrough

To put ourselves in the descriptive shoes of our user types. We defined key tasks that the users would want to accomplish at the site. When running through the website keeping in mind, each persona gave us a few key insights as to issues within the site and then, based on the way those issues were encountered, we provided suggestions on how to alleviate the issue/s.

User Profile

New User

Alex, 27, Maker/Designer/Student

Scenario

Alex received an invitation from his friend to join a new social network called Ello. Fed up with Facebook’s privacy, business practices, and intrusive ads, Ello became a great new platform for Alex. He is interested in exploring the main functionality of the site, and seeing what items/features are similar to Facebook and similar platforms.

Tasks

  • Create a profile from invitation
  • Edit your profile
  • Learn how to troubleshoot/use the site
  • Find new users
  • Blocking a user
  • Find features list and see what Ello is doing to finalize the build
  • Invite New Users

2. Existing User

John, 28, Designer

Scenario

John is an aspiring artist, a designer. He is 28 yrs of age and an avid photographer too. He loves traveling. He travels once in two months and loves to blog about it. He works as a full-time graphic designer at a firm. He is very imaginative too. In spite of his busy schedule, he wants to keep in touch with friends, other designers to keep himself posted about the latest news in the design field. He was using other social platforms previously but later got bugged with the advertisements showing up on his page. He wants to use Ello to post his designs work, photos etc. He owns a MacBook and a DSLR camera.

Tasks

  • Add a header image
  • Post a comment
  • Unfriend a follower
  • Search for a friend to follow
  • Quick logout

Some suggestions were made after the cognitive walkthrough

Lack of visibility of post comment button

Ah! I didn’t event see it, I tried pressing return key, still it didn’t work. Until I saw that tiny arrow.

Let’s see if you can find it?

Can you see the arrow?

Moving further we drafted a detailed usability test plan based on the expert review, cognitive walkthrough and heuristic evaluations. It can be found here.


User Testing

As this is a novel website design that does not follow all the standard platform conventions of any one social media site, most users wondered whether different words, situations, or actions meant the same thing. It was discovered that Ello is essentially a mash-up of Facebook and Twitter. This is why both new users and existing users had issues navigating throughout the website. They did not know whether they should follow the platform conventions of Facebook or Twitter.

Findings from the user testing pointed out following:

Lack of Visibility in Icon Design

Icons which are presented with low contrast were overlooked by users throughout the site. Users only noticed them when scrolling over the icon itself or when double-clicking.

Lack of Consistency and Standards

Users experienced issues discovering the meaning of the Notification icon, formatting bar, toggle buttons labels, and difference between adding and inviting users.

Lack of Confirmation Prompts

Confirmation messages are not supported for functions as ‘Friending’ and ‘Unfriending’. This caused confusion for most users.

Unclear Term Usage

Most users were unaware of the word usage of ‘NSFW’, ‘WTF’ or ‘Noise’.

New users rated Ello with a SUS score of 57.0 and existing users rated Ello with a SUS score of 48.0.

User Demographics

We tested ten users of which five females and five males. Their ages were between 20 and 30. The level of education ranged from undergraduate to Ph.D. Five were at an undergraduate level, Four were in graduate level and a Ph.D. student. Four users used social networking sites very frequently. Fewer used it occasionally and one of the users used it rarely. All of them were familiar with Facebook. Most of them were aware of Pinterest, Instagram, LinkedIn, and Twitter.

Use of social network from both Novice and Existing users.

Quantitative Findings

Time on Task

New Users

Of all the 5 new users we had for user testing, it can be observed from the graph that the average time on task for most of the participants is same. But for task 5 “Write a post about the weather in Indy, attach a link to one word and bold another” it can be seen that participant 4 and 6 took more time to figure out how to complete the task.

Time on Task for New Users

Existing Users

Of all the 5 existing users we had for user testing, it can be observed that participant 2 and 8 took more time than the rest of the participants. Also, participant 2 seemed to have taken more time in almost all of the task, except for task 3 and 9, which shows that participant 2 had trouble performing tasks on the site, except for “Unfriend a follower” and “Block a follower” which are placed in the same section of the website.


Qualitative Findings

Qualitative findings were captured after each task, the users were three questions as follows:

  1. How was that process for you?
  2. Did you find any part of that task cumbersome?
  3. Is there anything you would change?

Users made several comments while the testing was in session and we were able to gather excellent feedback.

Highlights from the feedback!

Task 2: Find help and documentation

“Wait, does WTF mean “what to find”.

Task 3: Find a new user and add them

Can you guess where the search bar is?
“I don’t see the search bar, why is it not working? There is a different layout on the search engine.”

Task 9: Locate your notifications

The user thought the noise is the notifications.
“It is here [Noise]. I know it is supposed to be one of the main icons, which is one of the biggest ones. I think that Noise means something to disrupt you, just tell you something that is going on.”

Overall, we were quite happy with the feedback we received from the users. We had also informed the Ello team about our work. Once the complete report was complete we sent the report to the Ello team. And today, if you compare the findings you will see that most of our recommendations have been implemented by the Ello team.

Detailed report can be found here, download

Presentation with highlights can be found here, download


Lessons Learned

This was my first project after coming to the HCI Masters program. I learned to work in the team.

  • Learned how to conduct a usability test and I was able to successfully conduct two user tests and observe two user tests
  • Understood the Neilsen’s Usability Heuristics and applying them to a website or application
  • Learned how to perform a cognitive walkthrough and create personas to evaluate tasks.
  • Learned how to perform a quantitative and qualitative analysis.

It was an excellent learning experience and the feeling of sharing the work with Ello. And see those changes happen was itself a great feeling of achievement. I hope you enjoyed reading this!

P.S. This project was a part of class project from 2014 at Indiana University, Indianapolis


Thank you for reading. Feel free to comment, offer feedback and your thoughts.