Designing for Fun and Expression: Placing Video Game Controls in the Palm of Your Hands
For those who have a Leap Motion Controller and want to try this out, our code is available here : https://gitlab.bucknell.edu/zp002/design_for_fun_repo
— Introduction
As kids, many of us grew up playing computer games, and became used to navigating by clicking a button or the mouse. In 2007, the first iPhone was released and over the past 10 years, there has been a rise in touch-screen games, apps, and phones. What was once a revolution in how users interact with technology has now become the norm and has ultimately impacted users’ level of engagement with certain games and applications.
For our “Design for Fun/Expression” project, the idea was for our team to build an interface over a pre-existing game or application and redesign the controls to create a new experience for users. This interface would prioritize being fun and allowing for a new expressive way to control or interact with the chosen game in order to maximize the users’ level of enjoyment.
Using a Leap Motion Controller (LMC), our team chose to remap the controls on two popular mobile games “Doodle Jump” and “Paint Hit”. The former is normally controlled by tilting one’s phone left or right to guide their character as it automatically jumps upwards, striving to stay on the platforms while avoiding the obstacles. In regards to the latter, rather than having the user interact with the game through a character, “Paint Hit” allows the user to have more of a one-on-one interaction with the game. In the original game, the user is to click on the screen to shoot a paintball at the rotating circles but with one catch; they cannot hit the same place twice.
A key reason for choosing Doodle Jump in particular, was that it’s a popular game that many of our peers were already familiar with. That being said, we thought it would be interesting to observe how our peers’ past experience with the game would affect their experience with the new way we’ve revamped the controls as well as their level of engagement.
Overall, we chose these particular games for two primary reasons:
- The challenge of both games depended on its controls and the skills of the player to use the controls
- The controls were able to be transferred to hand motions — without having to account for precision — while maintaining the original challenge
Our final products were two Java programs that allowed a user, through a LMC, to play Doodle Jump and Paint Hit with their hands, controlling the character by tilting their hand left or right for the former, and for the latter, controlling the shooting of the paint-balls by opening and closing their hand.
— Preparation
We decided to undertake this task by using a LMC since we were provided with one by our professor, though we were not required to use one. We tested the LMC on our laptops to ensure its functionality. While the setup was a quick process, we needed to become more familiar with the Leap Motion technology and get a sense of the accuracy of the LMC through its built-in playground that allowed the user to move their hands around and play a couple of simple games like fishing.
— Brainstorming
Our first task after selecting our tools was selecting what software application or game to use. We decided that a game would be the most effective way to create a fun and expressive control scheme and allow for the most interaction for the user.
Several of the games we considered before coming to our final choice:
Our criteria for choosing games was based around easy access to an online web version, variety of controls without losing the challenge of the game. We also focused on the way the game played. Games that were challenging because of their controls were perfect candidates. Ninja Jump and Subway Surfers were the first to be taken out because they either did not have an online web version or the game was too similar to what had already been done (i.e. Temple Run).
- 2048
While we were drawn to the simplicity of 2048, the challenge for this game, much like chess for example, lies in the player’s mind rather in how well they can move pieces around, and therefore would not be conductive for an expressive experience.
2. Surgeon Simulator
Surgeon Simulator became an option because of its uniqueness, but we eventually decided against it because it took too much precision from picking up the tools to convert into controls. We felt it would be too difficult for users and it could potentially take away from their enjoyment of the game from having to be so precise.
3. Fruit Ninja
The simplicity of Fruit Ninja and our familiarity with this popular game were key reasons why our team considered this as an option. However, many other groups were considering this as well and we wanted our game to be unique.
A big aspect of our brainstorming process was what is called “Wizard of Oz testing” in which we had one user pretend to play the game using gestures. Another tester would look only at the user’s gestures and press the controls accordingly. This type of testing allowed us to get a feel for how well our potential gestures worked, as well as how responsive they could be for the game.
After many more rounds of brainstorming, we eventually decided on Doodle Jump and Paint Hit. Our team felt that these games were perfect candidates for the type of experience we were trying to provide as they fit all of the criteria we required.
— Creating The Motions
While remapping the controls on Doodle Jump, our team debated around which gestures to include. Gestures were a key aspect of this project, and we wanted to use gestures that would be natural for the player to make and use constantly while they played.
Paint-Hit: Gestures
Paint-hit has a simple premise, the goal is to throw paint balls at a rotating platform, and the only rule is you cannot paint on the same area twice, the game makes this challenging by rotating the platform at different unexpected speeds. Players must time their clicks carefully to make it through the levels.
We first thought about mapping finger swipes or arm swipes to the clicking option. We tried to program and prototype different gestures to eliminate the discrepancy between “evaluation and execution” as mentioned by Andy Ko in his article Hands. Then, after a few trials of prototyping, we felt that it is not very intuitive since it doesn’t match the action they actually perform when playing the game.
In addition, because the game was designed to keep playing for higher levels, players have to put their hands up for a long time. According to Antti Oulasvirta in his twitter thread, a lot of the criticisms of AR/VR came from the shoulder pain for keeping arms up. Therefore, we decided to use fist/extended fingers to represent throwing the paintball. First of all, this is intuitive since it mimics the action of throwing the paint in the game. Secondly, players could have their elbows on the table, which adds more comfort to their interactions with the game.
We adjusted a sleep time between press and release, and a few milliseconds between frames to take human motor reflex reaction time into consideration.
Doodle Jump: Left and Right Tilts
Once we finalized our gesture of using tilted hands, the first challenge to overcome was calculating the differences between the Y-coordinates of the leftmost and rightmost hands. If the difference is above a certain threshold, we press the left or right arrow accordingly.
Left/Right handed people
However, once we went to prototyping, we realize that people have an easier time tilting their hands outward than inward.
Therefore, we adjusted the thresholds to make sure that the game worked well for both hands, but further optimization is still needed (like adding an option to select your dominant hand before playing.)
Doodle Jump also has a firing mechanic, triggered by holding down the space bar. We decided to use our fist to represent the fire option because a closed fist is similar to the round bullets the little alien is firing. We thought that would be intuitive and the least intrusive option as the player needs to keep tilting while they fire.
— Feedback and Possible Redesigns
After demo day, our team received feedback from our peers regarding the first iteration of our redesigned games. Users appreciated the fact that both didn’t require any learning in order to play as well as the responsiveness of the controls for each game.
On the other hand, our team could improve on the accuracy of the controls as they didn’t line up 100% with the movements of the users. For Paint Hit, the controls were extremely sensitive and it was difficult to shoot the paintball exactly when you wanted based on the gesture. Some found it confusing whether opening or closing your fist shot the ball and recommended we redesign the gesture itself for this particular game. For Doodle Jump, we lacked precision to a certain degree. When students tilted their hands, it doesn’t accurately measure how far the doodle itself actually shifts from left to right, which throws off the game.
— Conclusion
Overall, our team created a fun new way for users to experience two popular games, Doodle Jump and Paint Hit in a whole new way. While our peers gave generally positive feedback for our games, we still identified several areas of improvement if we were to create another version of our prototypes.
One aspect would be to make the games less sensitive to every movement. Sometimes, any shift would cause the games to react, and this was especially a problem with Paint Hit when students would open and close their fist to shoot the ball. To remedy this, we might change the gesture so instead of opening and closing a fist, which was confusing to some, we could adopt an in-air finger tap option. That way users could simply point their finger whenever they wanted to shoot and that increases precision while still allowing the controls to be responsive as well as decreasing the confusion component.
While we still had several areas of improvement, our team ultimately felt that we had achieved our goal of raising users’ levels of enjoyment through interacting with our redesigned versions of already-existing games.
___________________________________________________________________
Team Members: Judy Peng, Tiffany Cai, Bhagawat Acharya, Daniel Perez
