The Value of Published User Interface Videos for Intellectual Property Litigation
by Saul Greenberg saul@ucalgary.ca
I was lucky to attend the very first CHI conference in Gaithersburg, Maryland, and was a regular attendee for many ACM SIGCHI conferences afterwards. Over the years, videos were increasingly used by presenters to illustrate a paper’s ideas. The SIGCHI organizers at the time recognized the value of videos, and created a separate refereed video track to establish a historical record. These independent videos were published as part of the SIGGRAPH Video Review series (and later directly by SIGCHI), distributed to conference participants for a modest cost, and played both at the conference and in the conference participant’s hotel rooms on a dedicated TV channel. Later videos included video figures associated with papers. Other CHI-related conferences also started their own video tracks, such as the ACM UIST and ACM CSCW conferences. A historical description and access to these videos is found here.
As a researcher and educator, I found the ACM CHI videos — as well as many other interaction videos — useful for both understanding what people had created, and for use as an educational tool. Importantly, videos illustrate interaction nuances unfolding over time. While a picture may be worth a thousand words, a video can be worth a thousand pictures. Perhaps not surprisingly, I began collecting user interface videos from CHI and from other sources.
Somewhat later in my career, I occasionally consulted on intellectual property litigation (i.e., patents), mostly as an expert witness. For those who know little about this kind of litigation, here is a gross simplification of what happens.
- Patent. A patent describes an invention. The first part is the specification, which provides background, an overview, and details of the invention. The second part is the claims, which provide a specific list of what the patentees are claiming as the novel parts of the invention. Claims also sets the scope of where the patentees are seeking protection. Claims usually comprise a set of clauses called claim elements.
- Infringement Assertion. The patent holder can seek protection when it asserts that Company A has a product that infringes on one or more of the patent’s claims. That is, the patent holder believes that at least some part of Company A’s product does what the patent claims as being its invention.
- Lawyers get involved. Some very smart people are involved in figuring out strategies for fighting the case on both sides. This involves making motions to judges, doing discovery, taking depositions, etc. Lots of things happen here.
- Expert witnesses are hired by both sides. Experts are asked to gather and analyze evidence, state opinions, write reports and testify at trial. Their report and testimony is usually centered on two main areas.
Infringement vs. non-infringement: The patent holder’s expert provides evidence that Company A’s product does every element in each asserted patent claim. Meanwhile, Company A’s expert examines the product against the patent claims, and explains why that product does not do what the claims describe. For example, if even a single claim element is missing from the product, the product would not infringe. - Validity vs. Invalidity. Company A’s expert examines the patent claims against the prior art. If products or publications produced before the patent’s filing date are found to satisfy the claims, then those products or publications are evidence that the patent is invalid. Invalidity is either through anticipation (a single product or publication covers all of a claim) or obviousness (several publications or products can be combined to cover a claim’s elements, where that combination would be obvious to do by a person skilled in the art). The patent holder’s expert also analyzes those products or papers to see if there is a way to dispute that they cover the claim elements.
What may surprise readers — particularly those from a research perspective — is that the inventive claims of many user interface patents can cover interaction sequences that would never make it past a normal conference or journal peer review process. That is, they often describe quite mundane interaction sequences that may be simple variations from what came before it. From the patents I’ve read (including the file history which details the back and forth between the patent examiner and the patentee), the only reason (it seems) that some patents were granted is because the patent claims are just different enough — sometimes in quite trivial ways — from what the patent examiner was able to find in prior publications. In contrast to academia, where the threshold for a paper’s acceptance is that it should make a significant contribution on some level, this is not necessarily the case with patents. It just has to be different enough from prior publications. Of course, this is not to dismiss all patents, as some do describe ideas that would certainly meet our academic bar.
The role of videos
Videos of systems in action can become compelling evidence as prior art supporting invalidity contentions.
Videos are understandable by jurors. Most patent cases are fought in front of a jury, which usually consists of lay people with little or no technical knowledge of what is discussed. Understanding print material provided as evidence is difficult for them. First, patent claims are written in legalese, which are typically hard for jurors to understand. Second, prior art used to support invalidity (such as other patents and academic articles) are written in technical language and often requires background knowledge to interpret correctly. To make matters worse, terminology used by the patent and the prior art often differs, even when talking about the same thing. While the role of Company A’s expert witnesses is to explain to the jury what the prior art is saying, the patent holder’s opposing expert will often refute it, which makes it even more difficult for the jury to understand the similarities or differences between the prior art and the patent claims.
A video illustrating how a system works often removes that obfuscation. The beauty of a good interaction video is that it is easy to see exactly what a user is doing, and exactly how the interface responds. Thus videos make it far easier for a juror to understand the prior art being presented and whether it is, in fact, good evidence of invalidity.
1. Videos show mundane details of an interface’s behaviour. As previously mentioned, a patent’s claim may be on very mundane interface aspects and details. To find evidence of invalidity in prior art, I (as an expert witness) have to search academic papers and product manuals for those mundane aspects, ideally finding one that provides a good description matching the claim and its claim elements. Yet mundane interface aspects are not normally mentioned — let alone discussed in detail — in academic papers because they are often incidental to what the paper is really about. Similarly, a product manual may not include those details (or may do so briefly) because the interaction flow detail would be obvious when using the product.
While CHI technical videos normally concern the creative aspects of an interaction idea, the fact that the interaction is captured over time also captures and reveals the mundane details of that interaction. The expert can then walk the juror through the video, showing how it matches the claims and its particular claim elements.
2. Videos humanize technical works. Many CHI videos are created by researchers and their students. They narrate and even act as actors in the video. While none of this matters legally, their appearance in the video humanizes the work. As an expert, I can talk about how the researcher shown in the video has a history and stature in the field, how they are well known to others in the field, and how they demonstrated the patent’s concept years before the patent was filed. Being able to see and hear that researcher makes that researcher tangible.
3. Videos are best when officially published with a publication date. Lawyers must prove that the prior art was available publicly before the patent’s filing date. This can be problematic if (say) a video was produced but there is no official way to prove when it was published, let alone if it was made available to the public. This is why the ACM CHI and related video tracks are so valuable — they are timestamped with a publication date, and were provably sold and shown to a wide audience at the conference.
Why you should produce and publish detailed videos
Intellectual property cases may be far from your mind. But wouldn’t you be annoyed if you found that some company is suing another for millions of dollars for a patent whose claims cover ideas that you had written about or implemented previously? And wouldn’t you be annoyed at not being able to prove it simply because your system doesn’t run anymore?
Producing a video that shows the full workings of your system –not only of its main idea but of its mundane details that you think are incidental — may safeguard the above. Timestamping the video can authenticate its publication date. Making it publicly available — and associating that with an official timestamp — can also help. An excellent way to do that is to get your video accepted on a conference’s video track, such as the ACM CHI, CSCW or UIST video tracks. However, a simpler way is to check to see if your university or agency has a public research report archive (often maintained by the library) that you can submit to. These archives often let you submit reports that are officially timestamped and made publicly available through the web. If your own archive doesn’t accept videos, push them hard to make them do so.
Posting a video on a personal website or even on YouTube is far less effective, as it is much harder for lawyers to work with. They must prove that it was available by a certain date, and that the video posting somehow reaches the threshold of public disclosure. Yet this can be difficult in practice. For example, the opposing expert could perhaps state that merely posting a video on a web page is insufficient evidence to guarantee that the video was available publicly. Similarly, there is no guarantee that a date inserted in a web page is authentic.
Finally, one video stands out as an exemplar that tries to document the state of the art in our field: Brad Myers ‘All the Widgets’ (CHI ’90 Special Issue ). This video documents many of the ‘standard’ widgets produced by different vendors (e.g., Scroll Bars, Menus, Palettes, Command Buttons, Radio Buttons, etc.), illustrating not only their commonalities, but the many variations between them. It really was an exceptional project, and I wish others would produce similar videos contrasting various developments over time.
Example: The slide to unlock patent
Consider Claim 1 from European Patent EP 1 964 022, titled Unlocking a device by performing gestures on an unlock image, by Apple, Inc.
1. A computer-implemented method of controlling a portable electronic device comprising a touch-sensitive display comprising:
a. detecting contact with the touch-sensitive display (408, 1014) while the device is in a user-interface lock state;
b. transitioning the device to a user-interface unlock state if the detected contact corresponds to a predefined gesture; and
c. maintaining the device in the user-interface lock state if the detected contact does not correspond to the predefined gesture;
characterized by
d. moving an unlock image along a predefined displayed path on the touch sensitive display in accordance with the contact,
e. wherein the unlock image is a graphical, interactive user-interface object with which a user interacts in order to unlock the device).
This is Apple’s European version of its somewhat infamous ‘slide to unlock’ patent. If you have used early versions of the Apple iPhone, it tries to protect the touch-based slider that a user uses to unlock the screen. While the invention described in the specification is close to how the iPhone works, the claim is written in a manner that is broader than just Apple’s implementation.
Yet about fourteen years prior to the patent, Catherine Plaisant and Daniel Wallace produced a technical report and a video titled “Touchscreen Toggle Design” that described various touch-based sliders, and how they could be used to turn things on and off. (Even earlier papers by others also talked about touch-based controls, but did not cover the concept in the same level of detail, which limited their value as evidence). The video was published at SIGCHI in 1992, with one of the sliders illustrated below. While the paper is fairly clear, the video shows very fine interaction details not covered in the paper.
When played as evidence, the touchscreen toggle design video can be fairly compelling evidence that most of the claim elements in Claim 1 above were known well before the Apple patent. However, the Plaisant video did not explicitly cover ‘unlocking’: it mostly describing the very similar notion of turning things on and off. This may seem to be a trivial difference, but even that difference can matter. Fortunately, it can be overcome by using the video as part of an obviousness case, where it was combined with a pre-existing phone that implemented touch-based unlocking via a sliding gesture on the screen.
I was involved in the ‘022 invalidity case, which was fought in the UK. Partially based upon the Plaisant video, the ‘022 patent was found invalid. It was also found invalid in Germany.
Of course, a video is no guarantee of success, as it is up to the judge and jury to decide upon the evidence presented. Years later, a somewhat similar Apple patent was fought in the US between Apple and Samsung. In that case, the jury found the patent valid in spite of the evidence. A Federal appeal court did later overturn the jury’s verdict and found the patent invalid. However, an even later appeals court reversed that decision.
Addendum, and nothing to do with videos. The image below is a typical unlock mechanism used to lock/unlock an airplane’s bathroom door. Make of it what you will in terms of how you would rate the inventiveness of the slide to unlock patent.
