Annotation Tool Part 2: Retrospective on the Build

This article is Part 2 of our series on the Sequence Annotation Tool. If you’d like to see our objectives for building our own Annotation tool, read it here: Annotation Tool Part 1: Why We Built Our Own Tool.

Bounding box annotation using Sequence Annotation Tool

This article is a look back on our Annotation Tool build. We learnt a lot from it and thought we’d share our process. Here we’ll take a look at what we built and the insights that drove our decision making.

Let’s start at the beginning.

Version 1 — The Prototype

“The best tool is the one that does the job.”

We released Version 1 with minimal but essential features. Our intention at this stage was to serve our earliest customers. We were still exploring the viability of our annotation line of services and wanted to go slow and steady.

We started with:

1. Create new polygon/bounding boxes point by point
   An obvious necessity. No polygons + no bounding box = no annotations.
2. Move existing points along the path
   We knew we’d require some flexibility and leeway for errors. Being able to adjust and tweak the shape of the path would be basic.
3. Delete existing shapes
   Errors are inevitable. We wanted a tool that was forgiving enough for the user even though it was just an MVP.

We did not include:

Zoom and pan
We knew intuitively that this would be something our contributors would need. However, it would have extended the build and delayed our time to start the project. We ran with the browser zoom and kept our fingers crossed. 🤞

The primitive version of our Annotation Tool 😅

Thankfully, the outcome was that the annotation tool worked for what we required at the time *collective sigh of relief*.

However, a few frustrating factors quickly became apparent. We ran some usability testing with our contributors, and we gathered an enormous amount of feedback for Version 1. The pain points were clear.

The feedback collected highlighted a few issues:

1. Initial points on the path were too large when zoomed-in making fine-tuning nigh impossible.
2. Inability to click through a point to the point beneath it to close the polygon path.
3. Zooming in and out using the browser did not produce an image that scaled proportionally to the window.
4. Lacking tools to reduce errors — ‘undo’ and ‘delete’ points were unavailable making it difficult and tedious for contributors to correct mistakes — often resulting in the need to entirely delete and redraw shapes.
5. A few of our contributors needed tablet support (annotating images using your finger on your tablet is unsurprisingly a troublesome adventure). However, this group is a minority use case, so we needed to aid them but in a minimal but effective way.
6. The view was also cluttered and not responsive to browser dimensions.

All these insights allowed us to pinpoint our issues and understand our contributors better. We took all this feedback and injected it into our solutions for Annotation Tool Version 2.

Version 2 — Improving Precision, Error Reduction, and Usability.

Our goal for Version 2 was to reduce the likeliness of error occurrence and significantly improve the usability of the tool. Our insight being that even though we can’t control a person’s capacity to maintain accuracy, we could control the tools we provided that would help reduce or correct mistakes, and by doing so we could improve overall precision.

We approached Version 2 — armed with our insights. We wanted to create a power tool that was applicable for a general audience; so we tried to bring over some of the interactions and behaviours that we found useful in professional software. Therefore, a lot of the ideas around the new functionalities are referenced from other vector software like Photoshop and Sketch.

Version 2 feature implementations:

1. Zoom and pan
This was our most requested feature. Previously when zoomed in with the browser, the image would not scale linearly and you were not able to add more detail. Zoom and pan come as a pair because while working zoomed-in refines output, when zoomed-in too far, you lose a sense of the bigger picture. Being able to pan allows you to refine and maintain context of the surrounding image — this allows you to work more comfortably for extended durations.

Zoom and pan for image annotation tool

2. Visual cues
Cursor events that indicate different functionalities on hovering over different areas.

Cursors cues for: closing a path, adding a point, and panning the image respectively.

3. Add/Delete/Move point on a path
Being able to add or remove point detail on a path to get that curved edge just right was important for us. In addition, being able to undo a previous point without having to redraw the polygon was a basic necessity.

Adding a point to a polygon annotation

Deleting a point on a polygon annotation

Moving and repositioning a point on a path for a polygon annotation

4. Cursor snapping
There was a frequent point of frustration with overlapping points whereby it was difficult to find the starting point for the polygon and close the polygon shape. Being able to snap to the first point, reduces wasted actions and minimises unnecessary frustration.

Cursor snapping to close a polygon annotation

5. Hide annotation layer
This was necessary for contributors working on displays with lower resolutions. It reduces overall clutter in the view when working on images with multiple annotations.

Hide annotation layer

6. Annotation shape panel and highlights
The shape panel is a functionality that provides organisation and context of work done. It’s an overview of individual shapes that make up all the annotations within the image. In the shapes panel, you can choose to highlight a layer for review before you submit or delete whole layers that you might be unhappy with.

Highlight and delete annotation layer

7. Accessibility fundamentals and MVP gesture support.
Initially, we toyed with the idea of just having keyboard shortcuts instead of buttons. However, we concluded that visual buttons allow contributors to know these actions are possible and enabled functionality, especially on devices that don’t support keyboards.

We included gesture support ie. pinch to zoom and two-finger touch for panning on touch devices. These were fundamental to ensure that our improved zoom and pan functionality was available to all our users.

Overall, we consider Annotation Tool Version 2 a successful build. We made incremental improvements based on our learnings in Version 1 and they have shown to be impactful for our users in creating a more refined output.

We’ve already got some feedback from our contributors and we’re happy to report that the improvements have been well received :)

“I’m working on a project right now. I love the new tools :) Thank you!” — Grace

“The improvement really helps! The new feature makes it easier to annotate tight spaces as it allows overlapping points.” — Amie

Want to see the Annotation Tool for yourself? Take it for a test drive here: Annotation Tool Demo.

This article was written in collaboration with Andy Ngo.

Learn more about us at sequence.work. We provide annotation and classification outsourcing for data science teams.

Looking to work on tasks instead? Become a contributor here.

*Images used in the sample annotations are from Unsplash.