University of Connecticut: Digital Media and Design Independent Study
What is Kinect?
Kinect is a tool used for multiple applications as a tool that allows the user to be the controller through the motion sensing of the device. The software builds an image of the scene in front of the sensor that detects the position of players and builds a tracked skeleton based on their location and movements.
Setup
Aside from the Kinect Xbox One sensor, you will need a adapter for your PC as well.
With a background in video production, I am heavily integrated into the Mac workflow, so naturally I tried to make this process work with OS. Tip: It doesn’t… at least not well. If you’re a Mac-lover, it’s time to jump on the Windows train. After downloading all the software needed, my Macbook Pro still wasn’t recognizing the Kinect. I gave in and decided to go to a Windows laptop with still no prevail. I was a victim to unsuitable specs. Make sure you’re compliant:
Supported Operating System
Embedded 8 Standard, Windows 8, Windows 8.1, Windows 10
- Recommended Hardware Configuration
- 64-bit (x64) processor
- 4 GB Memory (or more)
- Physical dual-core 3.1 GHz (2 logical cores per physical) or faster processor
- USB 3.0 controller dedicated to the Kinect for Windows v2 sensor*
- DX11 capable graphics adapter**
- A Microsoft Kinect v2 sensor, which includes a power hub and USB cabling
So I ended up using BootCamp (integrated in Mac systems) to install Windows 10 on my computer.
Side Note: You need at least 50GB (at least on a 120GB hard drive) to use BootCamp. If you think you have the space, but notice “purgeable files” is taking up a lot of your drive, then copy over a sizable amount of temp files and delete them. This will delete much of your purgeable file space as well, free up your Mac and allow you to continue.
I also had to download all the software to make this work, as you can see below:
https://www.microsoft.com/en-us/download/details.aspx?id=44559
I’ve also noticed that a lot of this material isn’t updated and literature on the Kinect in general is scarce. Clearly a modified system is needed for this interactive concept to really takeoff. Read why Kinect failed here:
Time To Play
I wanted to make sure that the Kinect actually worked with my setup, so I loaded a few demo projects to make sure I had a reading from the sensor.
I initially began with the Kinect SDK, the most popular scene example being “rainbow man.” I was able to get this to work, but the scenes seemed really finicky to me so I pursed an alternative option. This Kinect SDK package may work better for a more experienced user, however. You can check this version out in the link below.
I used a k2-asset provided by Kinect v2 with MS-SDK based off of recommendations. I experimented with a few different asset packages, and this one was by far the most versatile and functions the best.
- Note: If you contact the developer directly at rumen.filkov@gmail.com, you can get a free educational download.
This asset package opens up interactions with the Kinect so you can focus on fine-tuning the experience. There are so many possibilities to explore including first-person, background removal, face tracking, gestures, etc.
Here, I was able to explore different types of interactions and how they were done. Different actions such as holding a grip, keeping a flat palm over an object, swiping, or simply colliding with an object can have different (or the same) results. I would highly recommend starting here to gain a sense of what what gestures can compliment what you will build on your own.
The scene that was most helpful to me was the collision demo since I was looking for a very simple interaction. My goal was to end a game if a falling object hit the person in the scene. If you don’t have a Kinect, you can download that prototype looks like here (only available to Windows users).
My personal process was to develop this game using a 3D game object (a cube) that the user moves with their keyboard to imitate the user. I then planned on removing the cube and use the skeletal Kinect reading of the user to interact with these falling objects (unfortunately I hit a couple of (did not save correctly bumps in the road and this was not as smooth of a process as I had originally intended).
In this particular game, I set it up so that the score increases 10 points every 10 seconds. If you are hit by a falling boulder or if you go out of bounds, the timer stops and the points stop, but it records, keeps, and loads a high score. The score, high score, timer, and restart texts are are located on the interface of the game. I built this so I could get the scoring and interaction code down and hoped to have a simple swap in with the Kinect model.
Kinect Game
My main goal here was to create interactions with the skeleton that the Kinect instantiates with objects in the game. To begin, I set up objects that you can touch with any part of your body and the objects will react in some way.
In order to begin the game play, you first need to raise your hand to initiate the body tracking. You will then see your body on screen, and when any part of your body collides with the skeleton, that object will respond. This is done by a simple script calling for a response from the game object on collision via the “onTriggerEnter” function.
*note: both objects will need colliders (one set as a trigger) and one needs a rigidbody in order for this function to work.
The Kinect has depth detection, so some objects are set farther back and forward in the game and you must walk in that space in real life to get to those objects.
You can play this game here.
If you don’t have access to a Kinect, on collision the boxes disappear, the little people jump, the pink spheres change color and the red sphere will move when you hit it. If nothing is happening, you may need to walk forward or backward to reach the object. Even though I was in a wide open, well light room, the skeleton seems a bit jumpy. If the joints aren’t green (which many of the times my were red), then you may have to move farther away from the sensor.
You can add up to six people in the game, and sometimes it gets confused over who is who, but overall is still a really cool technology to play with. If the body is not completely visible or lose clothing is worn, the software will estimate where the joints are. It also takes a little while for the Kinect to recognize you, so just keep that hand up and it will find you!
Falling Rocks Game
In this game, it is very similar to the build where you use the cube as your player, but instead I have the Kinect Skeleton now being the player. Instead of going to the objects, your goal is to avoid them.
A couple notes as the developer is that since there is depth projection, I need to make sure the user stays on a certain line in the “z” space. If they step too far forward or too far back they can simply avoid all the boulders and rack up that high score. Aside from a couple project rebuilds, I had to make a few changes in the code. One was, I couldn’t have the falling object destroy the Kinect Skeleton at all. This is because it would just take away whichever joint was hit and throw the Kinect script into a doozie. I avoided this by taking away the “destroy object” code and sending it straight to the “game over” function which stops the timer and the score, and replaces the high score if it’s the highest.
Moving Forward
The next steps I would like to take with this are to get better images with clear steps in the game, and project these images onto a wall. This way, the user is interacting with a projection instead of seeing themselves in skeleton form in the computer. I will need to make a big indicator (sometime like a flashing red screen) to make sure they know that they’ve been hit. I would also like to create several other projection interaction games so it becomes a small series.
There are so many different options and ways to interact with the Kinect, but not a ton of language on it. Much of this process is through trial and error, but the interaction is very entertaining component. More game updates coming soon!
Thanks for reading!!!