Integrating Gesture Controls into your Applications — Kai SDK
It has only been a little over seven decades since the first computers were invented. At the time, they were bulky and expensive but their spread was limited due to another major factor. User Interactivity. Scientists working on those computers, had to literally punch in holes onto boards to give binary instruction. Since this crude, cumbersome and a rather humble origin, our computers and methods we interact with them has come a long long way.
Gesture — The Future of UI
The human brain is hardwired to the real world. This means that it is not normal for us to interact with anything in 2D, especially not technology that we use so frequently.
To understand this better, we need to separate the concepts of Computer and its User Interface. A computer is essentially a processing device, a module that can do a lot of calculations very fast. It’s user interface on the other hand is how we interact with it.
The Kai allows us the liberty to interact with our computers, via a simple set of gestures. These gestures can be mapped to key board or mouse options. The limit here is your imagination. The Kai Control Center is shipped with a set of 8 gestures and 2 pinch actions, configurable by the user. The keyboard mappings can be configured separately for each application of the users choice.
Kai SDK
The Kai Software Development Kit interfaces with the Kai Dongle, allowing users/developers full control to the data being sent from the Kai Controller. These gestures and data can be interpreted and used as you choose, either to improve the performance of the existing device or add gesture controls to your apps, games and softwares.
The Kai SDK operates by reading from a web socket to which the incoming data from Kai is written. Any commands sent to the Kai is also sent via this web server. The web socket can be read from any programming language. Currently, we natively provide support for the following programming languages:
- C# (or any .NET based language)
- Node.js
- Python
Hardware Prerequisites
- Kai Gesture Controller
- Laptop/Desktop/Raspberry Pi
Software Requirements
- Kai Control Center
- Python 3.x
Setting Up Python Environment
Create a Project Directory and open a terminal inside the directory.
Step 1. Create a fresh Virtual Environment
$ virtualenv venv
Step 2. Activate Virtual Environment
$ venv\Scripts\activate
Step 3. Clone Kai SDK into Project Directory
$ (venv) git clone https://github.com/vicara-hq/kai-python
Step 4. Setup Kai SDK and Install into Virtual Environment
$ (venv) python setup.py build
$ (venv) python setup.py install
At this point, the python environment is setup with that Kai SDK.
Setting up Config File and Starting Web Socket
We setup a new file called ‘config.ini’ to store the web socket ID and access credentials.
# File-name : config.ini
# moduleID = "KaiApplication" # Name can be anything
# moduleSecret = "qwerty" # Leave as is[MODULE]
ID = "KaiApplication"
SECRET = "qwerty"
Create a new file ‘main.py’.
"""
File: main.py
Purpose : Kai Data Control Center (python)
"""import configparserfrom KaiSDK.WebSocketModule import WebSocketModule############# Global Varaibles #######
module = WebSocketModule()######### Evt handlers ############################ Functions ##############def get_config():
config = configparser.ConfigParser()
config.read("config.ini")
return configdef start_web_socket(config):
moduleID = config.get("MODULE", "ID")
moduleSecret = config.get("MODULE", "SECRET")
global module
success = module.connect(moduleID, moduleSecret)
if not success:
print("Unable to authenticate with Kai SDK")
exit(1)
Set Kai Capabilities and Event Handlers
The SDK operates on an event based format, to optimize performance. There are a few types of event generated in the SDK.
- Gesture: Makes the Kai notify when a gesture is performed.
- Finger Shortcut: A finger shortcut is defined when any finger (except thumb) touches the thumb.
- Linear Flick Data: Makes the Kai notify when a linear flick is perform by translating the hand in any one specific direction.
In addition to these, the Kai can also respond with data from the built-in sensors.
- Finger Position Data: Makes the Kai send the absolute positions of the 4 fingers
- Quaternion Data: Makes the Kai send the quaternions of the Kai position in the user’s hand.
- PYR Data: Makes the Kai send the Pitch, Yaw, and Roll (PYR) of the Kai position in the user’s hand.
- Accelerometer Data: Makes the Kai send the accelerometer values from the sensor.
- Gyroscope Data: Makes the Kai send the gyroscope values from the sensor.
- Magnetometer Data: Makes the Kai send the magnetometer values from the sensor.
Note: All of the above scenarios trigger events within the SDK which needs to be handled by the Application Code.
Additionally, to improve battery performance, the Kai responds only with data it has been requested for. The following code-snippet is for adding event handlers and setting up requests for data from the Kai.
# main.pyfrom KaiSDK.DataTypes import KaiCapabilities
import KaiSDK.Events as Events######### Evt handlers ###############
def gestureEvt(evt):
print(ev.gesture)def accelEvt(evt):
print(ev.accelerometer.x)
print(ev.accelerometer.y)
print(ev.accelerometer.z)############# Functions ##############
def main():
global module
module.setCapabilities(module.DefaultKai, KaiCapabilities.GestureData | KaiCapabilities.AccelerometerData) # Register Evt Listeners
module.DefaultKai.register_event_listener(Events.GestureEvent, gestureEvt) module.DefaultKai.register_event_listener (Events.AccelerometerEvent, accelEvt) time.sleep(30) # Testing Duration # Can be replaced on ESC break # Save kai battery by stopping requests when not Needed
module.unsetCapabilities(module.DefaultKai, KaiCapabilties.AccelerometerData) time.sleep(30) module.close()
Note: Multiple capabilities can be subscribed to and this will result in the module getting both data parallelly.
Run and Test Module
if __name__ == "__main__":
get_config()
start_web_socket()
main()
Finishing Notes
The output console will print the accelerometer data once every 9ms on an average. The Gesture will be printed only of the Kai is able to recognize a gesture.
The Python SDK allows developers to adapt the unlimited benefits of gesture control to their own applications. These applications may range from gaming and 3D modelling tools all the way to an active interaction medium on AR and VR devices.
The potential is truly unlimited…
In order to optimize battery life, the recommended way is to get this data only when required. An example of such a flow would be as follows. Let’s say we need to make an object rotate in real time on the screen.
- We would first subscribe for finger shortcut data.
- When a grab is detected from the finger shortcut data, we subscribe for either PYR or Quaternion data.
- We use the PYR / Quaternion data to rotate the object on the screen corresponding to the user’s hand.
- Once we detect that the grab has ended (from the finger shortcut data), we unsubscribe from the PYR / Quaternion data, to save battery on the Kai.
Useful Links
- Purchase Kai: https://vicara.co/home
- Kai SDK Python Docs: https://github.com/vicara-hq/kai-python
Web Socket Info:
- The Kai Published and Listens to the websocket at localhost on port 2203
- ws://localhost:2203/