Photon LightSaber Controller in VR

Ever since I was a kid, the LightSaber has been something that I’ve always wanted to build. With all the Virtual Reality and Augmented Reality trend this days, I wanted to learn how to develop software for it. I bought Google Cardboard when it first released last year and started exploring different apps. I’m amazed on how simple and easy it is to setup and run. I came across Unity3d game engine to create games for mobile including Google Cardboard. I started learning Unity3d which has vast amount of add-ons for Kinect, Oculus Rift, HTC Vive, Samsung Gear VR, Google Cardboard. I’m currently using Mattel ViewMaster VR which is based from Google Cardboard.

Virtual Reality gives immersive experience, but not having a way to interact with it, it gets boring after awhile. Came the idea to explore creating controllers in Virtual Reality. Just simple ones, like on and off. I started using Arduino via Serial port, then Bluetooth control, also tried Littlebits Cloudbit.

How about building a LightSaber controller?

Well, I found out that there’s Sparkfun sells an IMU module to Photon. I can use that to get the rotation for my LightSaber.

https://www.sparkfun.com/products/13629

The tutorial is easy to follow. Thanks Sparkfun.

https://learn.sparkfun.com/tutorials/photon-imu-shield-hookup-guide

After hooking it up, I wanted to have better data coming out of the IMU. So I researched into Kalman Filters. It can be to reduce noise data from the IMU.

https://en.wikipedia.org/wiki/Kalman_filter

Research some more!

I ended up finding this module in Arduino

https://github.com/TKJElectronics/KalmanFilter

I can use that but Particle.io has particular way of adding arduino libraries. Guess what? More Research.

So I ended up, creating a fork on the KalmanFilter project to incorporate to Particle. Here are some instructions how to create a library for Particle.io

https://github.com/spark/uber-library-example/#getting-started

Here’s the link to the Kalman Filter Photon Library

https://github.com/rlyle78/KalmanFilter

I modified the boilerplate code that Sparkfun provides with IMU and added the calculation needed for the Kalman Filter.

But how about the handle? How are you going to make it look like a LightSaber? Well, I know I still need to provide battery for the Photon. Then I saw this Lipstick-Size External Battery Power Bank. That’s my holder!

Now I got the “Controller”. I was able to connect it to Serial Port to my macbook. I’m getting data. This is getting interesting. Now what?

VR/AR requires it to be mobile. That means, deploy to mobile phone if I’m going to use it with Google Cardboard. Since I’m getting Serial Data on my laptop maybe I can just broadcast that or something. But I don’t want it tethered. There’s a reason why Photon is popular, Wireless IOT. Built in WIFI.

Time to research again. Thinking about it, it’s a WIFI and it’s Arduino based, there’s gotta be some libraries that it’s using to connect to the internet.

But what protocol? Should I use HTTP? Web Sockets? TCP? UDP? I need to get my data fast. So I decided TCP or UDP. Time to dig some more.

I found it in the reference and forums

https://docs.particle.io/reference/firmware/photon/#tcpclient

https://docs.particle.io/reference/firmware/photon/#udp

https://community.particle.io/t/receiving-udp-packets-on-the-photon/12825

TCP and UDP. Cool. I can actually send data.

How do I test it out. What’s the easiest way to test data sent from Photon? Well, I can do it in C#, create a network connection. I started also studying NodeJS this year. How about NodeJS? I found the documentation and this article.

https://nodejs.org/api/dgram.html

http://www.hacksparrow.com/node-js-udp-server-and-client-example.html

var dgram = require("dgram");
var server = dgram.createSocket("udp4");
server.on("error", function (err) {
console.log("server error:\n" + err.stack);
server.close();
});
server.on("message", function (msg, rinfo) {
console.log("server got: " + msg + " from " +
rinfo.address + ":" + rinfo.port);
});
server.on("listening", function () {
var address = server.address();
console.log("server listening " +
address.address + ":" + address.port);
});
server.bind(41234);
// server listening 0.0.0.0:41234

I started getting data from Photon via UDP to NodeJS.

But how to connect to my phone? I have Android, and I can’t use NodeJS on it.

I thought about it for some time, I started learning Unity3d this year. This is a good platform for creating games and visualization. Especially in VR and AR.

How about this? I said to myself. I can make my phone as Access Point … then connect Photon to my phone. Then I can now connect a Unity3d app to my Photon. Tada! It completes the loop.

My background is C# and .NET that’s why I chose Unity3d instead of Unreal. I’m hoping I can reuse some .NET libraries for communication. It’s now free to use networking stuff in Unity3d with the new version.

What’s good about Unity3d is that it has an asset store. Looking at the asset store, there are quite a few libraries that I can use but I need to buy it.

I found this article:

http://forum.unity3d.com/threads/simple-udp-implementation-send-read-via-mono-c.15900/

so I tried the UDP receive since I only want to receive data from the photon.

And It actually worked!

I connected the Photon to Unity3d via UDP sockets.

The important part of this project is the VR part. Game programming is new to me. So is Virtual Reality and Google Cardboard.

https://developers.google.com/cardboard/unity

I downloaded the Unity Package for Google Cardboard and open the sample app.

https://github.com/googlesamples/cardboard-unity

I started using the Google Cardboard Sample App. I incorporated the UDP code in C# that I worked on. Added logs just to get the data connection going.

Here’s the code that I used to receive UDP in Unity3d.

https://gist.github.com/rlyle78/84aa25133d1aaf0102cb

I deployed it in my android phone, ran the app…. It worked.

It’s still work in progress… Will update soon. Let me know if you want to learn more about this project. Follow me and add respect to this project.