Personalising the TV Experience using Augmented Reality
This article describes an exploration, by BBC R&D (Vinoba Vinayagamoorthy & Maxine Glancy) and IRT (Christoph Ziegler & Richard Schäffer), of using Augmented Reality (AR) technology to deliver sign language interpretations when watching a TV programme. We will be presenting a summary of what users thought of our AR prototype at CHI 2019, an ACM human-computer interaction conference, a session titled “AR/VR 2”, around 14:00 BST on Tuesday, the 7th of May. Full paper will be available in the ACM digital library.
Summary: We designed and evaluated three methods of watching a sign language interpreted natural history documentary. One was a traditional in-vision method with signed programme content on TV. The other two were AR-enabled methods in which an AR sign language interpreter (a ‘half-body’ version and a ‘full-body’ version) is projected just outside the frame of the TV presenting the documentary. We invited participants with hearing impairments to our labs, in the UK and Germany, to tell us which method of delivery they preferred. In the UK, participants were split 3-ways in their preferences while in Germany, half the participants preferred the in-vision method followed closely by the ‘half-body’ version. Full details of the design of the AR interpreters and our explorative study have been published at the conferences ACM TVX 2018 and ACM CHI 2019 respectively.
Context: In collaboration with industry partners, BBC R&D and IRT have been exploring how to customise and personalise the experience of viewing programme content on connected TVs by, in tandem, delivering additional (companion) content, to personal devices via IP.
We have been working with the industry to put in place the technical standards that will be needed to get devices to ‘talk’ to each other and synchronise media presentation at DVB and HbbTV. HbbTV technologies are a part of TV platforms such as Freeview Play and will enable audiences to benefit from these technologies. Our various demonstrators include synchronising content on a connected TV with content on a tablet or with a toy. We have also conducted user experience studies and collaborative projects, such as 2-IMMERSE, help us understand how we might create connected personalised experiences that might appeal to our audience.
So far, our investigations have focused on devices like a mobile phone or tablet. However, with the rise in popularity of optical head mounted displays, AR applications enable a way to make experiences that might spill out beyond the frame of a TV thereby extending the real estate of the TV screen and enabling personalisation of the media experience in an intriguing way. As part of a closed laboratory study, we wanted to investigate how AR can be used for the delivery of sign language interpretations to audiences with hearing impairments while they watched a TV programme. Traditional access services undoubtedly enhance our TV programme offerings for many users but they are an intrusive option which can’t be personalised or controlled beyond turning them on/off. How might we allow users watching TV in a group, personalise their viewing experience without imposing their preferences on the whole group? There is merit in being able to present personalised augments, like a synchronised AR sign language interpreter, to enhance the TV experience without using up the screen real estate of the primary viewing device.
Design Problem: How do we go about designing an AR sign language interpreter?
The design of traditional in-vision signage provided us with guidance on the creation, capture and presentation of our AR sign language interpreters. Several transferrable pieces of knowledge were identified from the UK-based industry regulatory authority Ofcom’s ‘Code on Television Access Services’, and prior work in Germany and the UK, in organisations such as the BSL Broadcasting Trust and the Deaf Studies Trust.
We settled on two designs — the ‘half-body’ and the ‘full-body’ AR interpreters. Full details of the design process and key principles we used to create the final designs are available in our ACM TVX 2018 work-in-progress paper. A summary of the guidelines are:
- The heads of AR interpreters have to be aligned to the top of the TV,
- The bottom edge of ‘half-body’ AR interpreter have to be aligned with the bottom of the TV; almost sitting on the TV stand,
- The feet of the ‘full-body’ AR interpreter grounded in the physical room,
- The direction of gaze of the interpreter has to be horizontally facing towards TV while in resting phase and has to be directly facing towards the viewer during the interpreting phase, and
- The interpreter needs to be positioned slightly overlapping the TV.
Method: We invited 23 participants (11 in the UK, 12 in Germany) to watch natural history documentaries in our user labs and evaluate three methods of delivering synchronised sign language interpretations — one traditional (picture-in-picture) in-vision method and two AR-enabled methods with the ‘half-body’ and ‘full-body’ AR sign language interpreters, presented through a HoloLens.
Findings: Participants were asked to fill questionnaires and we conducted a semi-structured interview to elicit their thoughts on the three methods we evaluated. In the UK, participants were split 3-ways in their preferences while in Germany, half the participants preferred the traditional method followed closely by the ‘half-body’ version.
Unsurprisingly, participants found wearing an optical head mounted display, while watching TV, uncomfortable. They also thought the field of view afford by the device to be too narrow to cover the TV and the AR sign language interpreter just outside to the right of the TV frame. Although, not all discomfort could be attributed to technical or physical limitations.
“Having glasses on your head for two hours? No way! And I can imagine, you put them on and everyone will look at you and they will recognise you are deaf. I don’t like showing the outside world that I am deaf. I feel too vulnerable”
— P10 (UK)
Despite the limitations, participants liked the idea of using novel technologies to solve a real world problem in TV watching and were open to the idea of using AR technologies to receive synchronous sign language interpretations in future iteration of these devices. Our qualitative data doesn’t reveal outright preferences between the three methods of sign language delivery presented but participants always had a clear favourite with clear reasoning behind their choice.
“She was sharper and she felt closer to you. It felt like I had my own personal interpreter in the room with me. I wanted to offer her a cup of tea.”
— P7 (UK)
We got a deeper insight into our participants’ perceptions and implications for improving the potential design of AR interpreters. Full details of our findings are in our paper. In summary, the importance of clarity of interpreter, maintaining the connection between interpreter and content, placement of interpreter relative to the content, and maintaining the balance between the interpreter overlapping over the content without obscuring the content was stressed. The underlying advantage in using AR technologies — the ability to tailor interpreters to the viewers needs — was seen as very exciting.
Although, we did not see a consensus on the preferred interpreter presentation, our results show that the demands made by deaf viewers on the sign-language service are very individual. Participants in our study highlighted the importance of control when it comes to personalising the experience to their needs.
Personalisation is key!
Paper citations
Vinoba Vinayagamoorthy, Maxine Glancy, Christoph Ziegler, and Richard Schäffer. 2019. Personalising the TV Experience using Augmented Reality: An Exploratory Study on Delivering Synchronised Sign Language Interpretation. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (CHI ‘19). ACM, New York, NY, USA, Paper 532, 12 pages. DOI: https://doi.org/10.1145/3290605.3300762
Vinoba Vinayagamoorthy, Maxine Glancy, Paul Debenham, Alastair Bruce, Christoph Ziegler, and Richard Schäffer. 2018. Personalising the TV Experience with Augmented Reality Technology: Synchronised Sign Language Interpretation. In Proceedings of the 2018 ACM International Conference on Interactive Experiences for TV and Online Video (TVX ‘18). ACM, New York, NY, USA, 179–184. DOI: https://doi.org/10.1145/3210825.3213562
