Civic User Test Session for the City of Miami Beta Site— Nov 29, 2018

This is a report written in detail of steps completed in lead up to and during Code for Miami’s Civic User Test Session conducted on behalf of the City of Miami to test the beta version of their website.

Planning and communications in lead up to the test

Two months in advance of our test session, folks from the office of the Information Technology Department reached out to plan a civic user test session for their evolving City of Miami beta site. Under the CUTGroup Miami, this and other additional tests in conjunction with the City of Miami have taken place over the last 1.5 years: read about one of those sessions here.

In advance to the test, we worked with the Information Technology Department to discuss what scenarios testers would be lead through and other logistics. We also secured our test location during one of the weekly Venture Cafe Miami events.

Three weeks before the test, an email was sent out to our proctor list from the CUTGroup Miami soliciting for help with the upcoming session. Two weeks before the test, we reached out to our potential pool of testers via email seeing if they would be interested in sharing feedback during the test in exchange for giftcards.

We averaged that given the space allotted to us and time from 6:30 pm — 8:00 pm, that we would have time for around 16 test sessions at 25 minutes each, with four proctors and one organizer present.

Goal

The primary goal of this test session was to see ease of use and navigation around the new beta site.

The Results of the Test

On Thursday, November 29th from 6:30 pm — 8:00 pm, Code for Miami members and folks from the Information Technology Department ran a civic user test session during that evening’s Venture Cafe Miami.

One scheduled proctor didn’t show up. Another, one showed up but thought they were only there to test. Two members of the Information Technology Department were on hand to assist with the testing. Out of the 16 scheduled testers, 13 showed up.

10 testers tested from tablets, with the remaining three testing from a laptop. With the exception of one on Safari, all testers used a Chrome browser.

Tester demographics

• Very tech proficient, background in computers.
• Retiree.
• Computer Training Specialist
• 33 year old moderate technology user
• Essence — Mother of one
• Miami Dade Resident
• Proficient in technology, 35
• Educator
• Computer Training Specialist
• Resident
• Joyful and creative
• Tech Savvy
• Young Tech Savvy — uses technology everyday
• Tech-Savvy

QUESTION #1 — Review the homepage and tell us about your first impressions.

Main observations:

• Almost all testers commented on the “nice” pop of colors.
• Three stated they were naturally drawn to index section.
• Most noted that it’s more user friendly to understand and access department and city services info.
• Two mentioned they like the larger font, while one said they’d like more to appear in the first fold.
• Most commented on the clean design.
• One found that the tabs appeared a bit crowded on a smaller window.
• Two mentioned issues with the search functionality.

When asked what they think this website does:

• Most testers answered that it was to help find information on services and events in Miami, specific to City of Miami gov.
• Only one tester mentioned that they thought the site could help them find information on permits, government officials, and ordinances.

Who do you think this website is designed for?

• Most answered that the site is for residents
• A couple pointed that it’s for professionals, business people, contractors
• One found there wasn’t any info for tourists so, “Not for tourists.”
• “Anybody that knows anything about computers.”

After reviewing the homepage, what is the first thing you want to do or click?

• Two mentioned careers and employment.
• Two showed interest in access to events at the City of Miami.
• Solid Waste pick up services schedule.
• Pictures of the administration people.
• Reserve a park.
• Trolley Tracker.
• Parking Information.
• Neighborhood information.

Question #2 — Show how you would attempt to obtain a permit to build a fence. What should you consider before you host your garage sale?

Actions taken learn how to obtain a permit to build a fence:

• 7 went to Building & Permitting section and selected apply for a permit, get a permit, clicks on garage sale permit
• 6 Searched for garage sale / permit, found apply for a garage sale permit.
• 1 attempted both clicking around then searching for permit before finding the application.
• Most didn’t know that you needed a permit for a garage sale. Consider eligibility, and that you need to do it 10 business days, and hold only 2 per year. One questioned “How would she know how many people were attending this garage sale. It’s not in your control.”

What documentation is required to apply for a garage sale permit?

• All testers found that they need proof of id, drivers license, proof of homeownership letter from the owner/ letter of consent from property owner, applicants information, and when the garage sale is going to take place.

Average ease of completing this task: 4.84 out of 5

Suggested improvements:

• 7 suggested no improvements
• Less scrolling
• Show category under Building and Permitting bucket.
• Search should be a little more visible.
• “There was a glitch when trying to request a permit search where once we put in a Zipcode, we found that the number kept coming up as the first number after putting in 2 numbers. This should be fixed.”

Question #3 — Find where to watch live commission meetings.

Actions taken:

• All testers clicks “Your Government” and then clicks on live meetings link
• Three click on City of Miami tv and finds that it doesn’t work/ is a legacy commission meeting page.
• Four testers are taken to old site when clicking on city officials
• One tester finds calendar for commission meetings but no link to further information

Is there a live meeting happening now?

• 1 Yes by way of clicking on full agenda
• 3 No “because a video wouldn’t load when clicked”
• 2 clicked livestream button and was led to black screen
• Most determined that there wasn’t a meeting, but took a while to come to that conclusion.
• Two found that there wasn’t one happening now through looking at upcoming meetings.

When is the next live commission meeting?

• January 10th 2019 9am.
• Scrolled through the legacy videos page, clicked on meeting group dropdown, found City Commission. Determined that it was December 13th.
• Looks the search and finds the schedule.
• Sees other events but not commission meetings.
• Finds it in the list on 12/13. Never found it on the new site. found that the next one is on January 10th, on the meetings landing page. Also went to archive page, which had a different set of dates.
• Most were able to find either December 13th or January 10th but found the dates were inconsistent or hard to find with other information.

What is the agenda of the next live meeting?

• Most clicked on agenda link and were led to old site with full list of meetings. They found that you could subscribe for updates but couldn’t find the actual agenda.
• Two selected the agenda and found it easily.
• One noticed the Agenda tab was there but it wasn’t for the city one.

Average ease of completing this task: 3.66 out 5

Suggested improvements:

• Indicate via a banner or something when a live showing is next or currently happening.
• One suggested that there are a lot of things related to the commission meeting, it should be on the home page.
• City of Miami TV seems like the place to put it or under city leaders or as its own icon.
• Two would not improve process.
• Make the navigation easier.
• Would put all the commission meetings on the same page.
• Four suggested that the agenda should show next to the meeting and go live links.

Question #4 — Determine if you can appeal a citation.

Initial actions taken:

• Most select Solve a Problem section, finds appeal a ticket, and selects the correct page
• Searches in search bar
• Building and Permitting, Appeals and Hearings — Request a Hearing in Code Enforcement (this is not the appeals service page). Could not find info on appealing citation, went back to appeals and hearings landing page and continued looking. Went back to Request a Hearing. Could not find this information. He went to search and found it.
• Four testers selected services available, and appeal a code enforcement before going back to click on Solve a Problem

Request an appeal hearing. Actions taken:

• 11 clicked on correct button and were able to fill out the form.
• Two could not find further information.

Average ease of completing this task: 4.6 out 5

Suggested improvements:

• Did not know what a code enforcement meant.
• Appeal a code violation should be included.
• 9 Would not improve process
• Two suggested changing the font and layout, too small.

Question #5 — How much does it cost to ride the trolley?

Actions taken:

• 4 Click on trolley tracker from homepage and easily found information.
• Search for cost trolley which redirected tester to trolley page. Found it.
• 5 clicks on transportation and find right section.
• Most felt the fact that it is free should be better highlighted.

Find the tracker. Actions taken:

• All testers easily found the tracker.

Alter the view of the tracker to make only the Coral Way route visible. Actions taken:

• Most testers had trouble selecting this. Noted that the font should be bigger and that it should be more responsive on mobile and tablets.
• Three were able to un-check all routes but Coral Way.

What is the estimated time of arrival for the Coral Way trolley to stop at stop 6 — SW 34th Ave. Actions taken:

• Most clicked on times tab and selects coral way route, clicks on stop 6.
• Three testers got stuck in a loop between the trolley page and tracker, unsure of where to go next.
• Three were unable to complete the task.

Average ease of completing this task: 3.84 out 5

Suggested improvements:

• Make the tabs on the left more obvious.
• Bigger labels.
• Close one menu when the other is open.
• Make the icons more responsive on tablet — larger and easier to click.
• Making the arrival times clearer by using actual times not minutes.
• Allow you to enter start end destination so you can get the estimated arrival time.
• It was hard to navigate and get back once you went to the trolley tracker.
• A non-tech savvy person wouldn’t know that the navigations, “Route” or “Time” should be placed on the top of the map. A place where it’s visible to a user that is not tech savvy.

Raw data — via Google Sheets

Conclusion

In this session, it seems that a bit hosted on the beta site still links back to the old site. While testers were able to complete most tasks with ease, there was a bit of jumping around to find the most pertinent information alongside other relevant details. Overall, they like the design and layout with the bold colors, larger fonts, and use of bullet points.

Next Steps

• Hold another test session once updates have been made, sometime at the end of Q1 2019.
• Assure variety in test pool for next session via hosting in a different location and/ or additional fielding during initial email survey by age, profession, zipcode, gender, etc.
• Secure payment method for proctors.
• Test from different browsers.
• Have additional laptops to test from.