Case Study: GradeML for WWDC 2019
Designing for Machine Learning and Drawing Recognition
Overview
Apple hosts WWDC every year to showcase what’s new in software and reveal upcoming features for Apple products. During my internship on the Machine Learning/AI team, I had the opportunity to work with software engineers to design a few demo applications that would be used to showcase new and improved machine learning features.
This case study will focus on only one of the demo applications and is condensed due to a very short timeline from design to production.
The Challenge
The software engineers recently created a way for users to capture pictures of test papers and grade them utilizing the Apple Pencil. The main goal of this demo application was to showcase the ability for the iPad to recognize handwriting and keep score of the test that is being graded.
Key Objectives:
- The handwriting should be converted to a digital form and the total score should update.
- Users should have the ability to manually override what score the machine learning algorithm determines from the handwriting.
Design Objectives:
- Keep the design minimal as to not distract the user from the canvas area that they will have the most interaction with.
- Create a way to correct mistakes made by the machine learning algorithm.
Design
User Interface
The UI is designed in a way that keeps the focus on the canvas. Marks created with the Apple Pencil will be automatically converted into a digital form and the system will keep score. The left panel can be toggled to take advantage of the full display size.
Handling Errors
Just like humans, AI makes mistakes too. When this happens, users should be able to correct it. Erasing the mark and writing it again is one option. Another option is tapping on the number where the error icon appears and using the scroll wheel to adjust the score to what it should be. Using the second option will help the system learn how to determine the users handwriting to avoid errors in the future.
Adjustments
There should be three methods for users to make adjustments. One, select an item on the list and use the scroll wheel (left). Two, tap a mark to move, copy, or delete (right). And three, use the erase functionality of the apple pencil.
Feedback
After finalizing the design and collecting feedback from the software engineers, the main concern was the short timeline for development. At first, it seemed that we would need to scale back the features in the design to meet the quickly approaching deadline. We discussed some options on a different design that would still showcase the ML features without focusing so heavily on the interface. After further collaboration with the software engineers and discussion with the project manager, they decided to put in the extra work and fully commit to developing this design as it represented a more refined application.
View a short clip of the design on stage at WWDC by skipping to 21:13 in the video below. (Keep in mind that it was not prototyped on stage due to the amount of content covered in the full video)
https://developer.apple.com/videos/play/wwdc2019/420/
