Dec 17, 2021
Pepper: Object Sharing with Holograms
Created in collaboration with Brad Lei, Daniel Rapport, Emily Zhang, Grace Chen, Ian Gonsher, Rohan Upadhyayula, Sahil Bansal, and Seth Eiseman.
In the wake of the Coronavirus, humans across the globe find themselves relying on virtual communication more than ever. With work from home practices quickly gaining popularity, it is important to analyze the technology fueling it to look for possible areas of improvement. While video conferencing technology offers many advantages such as real-time video communication, chat functionality, and collaboration, it also carries some major constraints.
Project Statement
‘Information sharing’ over video conferencing is a tedious procedure that relies on verbal communication and image sharing on flat screens, effectively dampening the overall telepresence experience. We sought to explore hologram technologies that would allow for a more realistic sense of telepresence. We aim to devise a new product that will accompany teleconferencing to allow for an immersive, multi-dimensional ‘object sharing’ so both parties can discuss objects of importance.
Object Scanning
To share objects, use the Pepper mobile app to scan the object, and convert it into a 3D model, making it ready to export as a hologram.
Object Sharing
With Pepper, people are able to view holograms of 3D files using the Pepper Mobile app and the viewing box. To create this hologram, we utilize the illusion technique Pepper’s Ghost. To display a hologram, open up the Pepper app and place a phone on top of the viewing box. The app will then sync with the current Zoom meeting and display the object for the other participants. We chose the phone to be the main source of the holographic image since it is an accompanying experience for the current meeting. This allows the meeting participants to still converse with each other and still interact with the object in real-time.
Final Storyboard
Inspiration
Hologram imaging technologies come in a variety of formats, ranging from lasers to fans. With cost, application, and scenario in mind, we decided to use the illusion technique pepper’s ghost to create our hologram. By placing a clear pane of plastic at a 45 degree angle to a screen, we can create a realistic hologram of the screen’s images on the plastic.
While our project was originally aimed at utilizing pepper’s ghost to improve the telepresence experience (providing a more realistic sense of presence for individuals who are calling one another), we found there were many issues with this initial application.
The problems we ran into mainly involved finding a suitable way to capture and represent the two video callers. For example, in order to create a realistic and compelling hologram of a video-caller, we wanted to make the projection as large as possible. Most users join conferencing calls on their laptops, but if the laptop was used as the main screen to create a hologram, it would be placed oddly and would no longer be available to use for any other work. We would need a separate camera to capture both callers because of the laptop placement.
Additionally, because the holograms require dim lighting to work well, we ran into issues with properly capturing the faces of the callers; the main purpose of using pepper’s ghost was to enhance the interaction between two callers but this technology’s requirement for a dimly lit environment contradicted this purpose.
Therefore, we pivoted our project to focus on object sharing, which we identified as another major flaw in modern video conferencing. We believe that in calls where users share a goal of synchronously viewing a 3 dimensional object for presentation or critique, the addition of an accessory to display this object in a realistic form would drastically improve the video calling experience.
Because phones are compact and are always carried around, we decided to use them as the screen to create the pepper’s ghost hologram. We tested a variety of setups for pepper’s ghost, including cones, pyramids, free-standing screens, and boxes — eventually settling on the box for the size and quality of holograms produced.
The Creative Process
Initial work
For our initial work, we created several low fidelity prototypes of different forms of hologram screens.
Cones
Inspired by research projects including a Google paper posted in 2017, we started by exploring a cone shape for the projection. With the help of our professor, this prototype shown above includes a thin screen that can slide into a channel in the central 3D printed clip-piece and form a cone that can be easily placed onto a screen.
Pyramids
The pepper’s ghost setup for devices like the cell phone stereotypically includes a pyramid: a square pyramid with no base that stands upside-down on the screen.
We initially aimed to use a pyramid to create our hologram because it provides users with a holographic image from 4 sides, which enhances the dimensionality of the illusion. This initial iteration involved placing a free-standing pyramid made out of acrylic on a phone to produce a holographic image.
Free-Standing Screens
When experimenting with the different forms to create pepper’s ghost holograms, we noticed that none of the forms were easily collapsible or transportable–they all consisted of 3 dimensional shapes that would be difficult to carry around. When thinking about the convenience and portability of creating a holographic device, we wanted to separate the plastic screen (used for the projection) from any bulky support material. Therefore, we created a simple initial prototype consisting of only the plastic screen.
Boxes
To initially test out the box form, we created a crude box using cardboard, tape, and clear plastic. In comparison to other forms, we felt that working with a box (by placing the phone screen on top and having an angled screen within the box) produced the best holograms. This form hid the screen of the cell phone from plain sight by supporting the phone at the top of the box and created a clear image because the closed environment blocked out more light than the other forms.
Second Iteration
Cones
Though the cone is theoretically supposed to enhance the horizontal viewing angles, in our experimentation we found the opposite to be true because the more the viewer moved horizontally, the more the image distorted. This can be mitigated by ‘warping’ the image projected onto the cone, but we felt that that process was tedious and was not worth the trouble of implementation for the slight benefit it provides.
Pyramids
While our initial model of the pyramid provided a decent holographic projection from four sides, we still found that there were issues with the prototype. Namely, we could see the image on the phone screen below the pyramid (which ruined the illusion of a hologram) and this particular set-up required a dark environment to produce a clear hologram. Therefore, in our second iteration, we tried inverting the pyramid and placing it in a box form so the phone would sit on top of the pyramid and more light could be blocked out.
Although this form provided a relatively realistic hologram from several angles, we found that the holograms were too small–for this model to work, we would need to edit our images to fit four of them around a center point on a single phone screen.
Because this form of screen provided users with a view of the hologram from several different directions, we considered using it to present multiple sides of a 3-dimensional object. In the video above, we showed four sides of a helmet and intended to rotate the screen and pyramid to provide users with an all-around view of the project object. However, we came to realize that this method of projection ruined the illusion of having a 3-dimensional hologram because the edges of the pyramid disrupted users when viewing between sides. We brainstormed other methods of displaying the entirety of an object (such as taking 360 degree videos or scanning the object), which is reflected in our final iteration. In the end, we decided against using a pyramid-shaped screen for our object-sharing product.
Free-Standing Screens
Although our initial prototype of the free-standing screen was easily transportable, it only worked in extremely dark lighting and didn’t attach well to most phones, as it required the phone to have a stand to complete the setup. However, we were committed to moving forward with a design that allowed users to bring a hologram-projecting device around with them.
Therefore, we incorporated a phone stand into this iteration and fleshed out a user flow to consider how individuals would store and interact with this product. We felt that the free-standing screen form had a major flaw: the phone blocked the majority of the user’s view of the hologram. Although it was simple and convenient, this form made hologram viewing uncomfortable and tedious.
Boxes
Out of all the forms we tested during our initial work, the box seemed to provide the clearest holograms. Therefore, we wanted to move forward with two versions of the box form: a solid, more permanent version that users could keep in their workspace and an easily collapsible and transportable version for users to bring around on-the-go.
For this iteration of the box form, we also found a reflective film made especially for Pepper’s Ghost holograms to apply to our plastic screens, which had potential to improve the clarity of the hologram.
Permanent Box
For the more permanent version of the box form, we played around with and created higher fidelity prototypes with two different types of walls.
The first type we tested was a solid-walled box; we essentially modeled and 3D printed the structure that we first tested out using cardboard. We found that this form was quite similar to existing hologram box products and that using the reflective film didn’t improve the experience of viewing the hologram. We set aside this iteration because of these flaws.
The second type we tested was an open box with no walls. We felt that this design was appropriate because the closed-box environment restricted the viewing angle to a minimal space directly in front of the box. With open walls, users can see the hologram in a less restrictive environment. When coupled with the reflective screen, this design generated a convincing hologram–the lack of a backing wall allowed the background to show through the screen, which essentially superimposed the hologram in the user’s physical environment. The reflective screen also allowed the hologram to be seen in brighter lighting than with other prototypes.
Portable Box
For the portable version of the box form, we came up with a couple of designs that would allow us to collapse the viewing box.
The designs were a) using flaps like in a pizza box, b) having the sides connect like jigsaw pieces, c) having the sides connect via slots and tabs, and d) having open sides supported by sticks at the edges. There were some issues that arose with each of these designs. The pizza box form became too thick as it required a double fold on certain faces. The jigsaw, slots, and sticks box forms had too many separate pieces that could easily be lost.
Therefore, we decided to combine these different designs and have a box that used flaps, slots and jigsaw pieces and could collapse down into a single piece. Our various designs can be seen in the sketches above.
Final Iteration
For our final iteration, we decided to move forward with the box form, as it provided the cleanest and largest hologram out of all the forms we tested. We maintained the split between our portable and permanent versions.
Permanent Box
After deciding to pursue the ‘wall-less’ version of the permanent form, we decided to create a ‘final’ version of this design. Shown above, this design features three transparent-PLA pieces including the left and right sides of the box and a small kickstand. The user can easily snap the two sides of the box together, slide an acrylic screen into the slots from the top, and then use the kickstand to appropriately adjust the viewing angle of the hologram.
Portable Box
After modeling several forms in Illustrator and testing out a variety of materials, we settled on the collapsible box form shown above, created using cardboard and the plastic screen. As shown in the video above, this box folds down into a single rectangle that can be easily slipped into a bag to be carried around.
Our two box forms explore the tradeoffs between portability and visibility. While the permanent box form provides users with a more immersive viewing experience by superimposing the hologram on the background of the user’s environment, it cannot be easily collapsed or transported. Due to its permanent nature, users will likely use this box form in an environment where they can dim the lights and enhance their viewing experience. On the other hand, while the portable box form is easily collapsible and transportable, the viewing experience it creates is more limiting. Because users may need this product in a public space, we created this box to have black, closed walls to block out as much light as possible. This creates a more closed viewing angle and the additional light can reduce the quality of the hologram. However, the portable box form is a great cheap alternative to the permanent version.
Reflection
Looking back on this project, our team went through several different prototypes and countless iterations to arrive at our final product, Pepper. There were several key decisions we made that helped shape the path we took.
We believe that we made the right decision pivoting the project focus from telepresence to object sharing as it allowed us to make a better argument for the use-case of such technology. Pepper’s Ghost was not a good fit with video conferencing because the holograms it produces are not at a high enough quality that it outweighs traditional video calling. In addition, when considering user input we realized that a user would be unable to use the camera on the device used to generate the hologram and so would need a separate camera. We started to think divergently about different use case scenarios where a hologram would actually be useful. As we thought about the telepresence experience, we realized it lacked the ability to easily share objects, so we decided to work towards improving the object sharing experience of video conferencing.
Future Work
Near the start of our iterative process, we had worked on a few prototypes to produce holograms using laptops and tablets. However, they were discontinued because perching a laptop in a way that was unconventional felt unnatural and could be unnerving for the user. In the future, we could continue or expand on these efforts to create larger holograms, which would help expand beyond the phone and open the avenue towards more use cases.
For example, holograms could be applied in a personal setting for users to view clothing, accessories, or other products in a 3-dimensional form for online shopping. Holograms could also be used to display art or allow for user interaction with holographic sculptures. Holograms provide an amazing, novel experience for users in any setting — the possibilities are endless.
