Videogame control through EEG (part 1)

This article is the first of a series that covers the process to implement an Emotiv headset as the input device in a video game made with Unity.

Requirements

EMOTIV EPOC+

Beforehand

It is necessary to understand some concepts first to start developing with an emotiv headset.

The way you connect and communicate to any Emotiv device is through an API named Cortex, it is built on JSON and WebSockets and can be accessed easily from a variety of programming languages and platforms.

The cortex API has a protocol made up of 3 building blocks:

WebSockets: which is a communications protocol for a persistent, bidirectional connection from a user’s web browser to a server.

JSON: It is short for JavaScript Object Notation, and is a lightweight data-interchange format. It is easy for humans to read and write and easy for machines to parse and generate.

JSON-RPC: it is a lightweight remote procedure call (RPC) protocol, basically it is a given structure and rules used for sending requests and receiving responses over many various message passing environments, like the websockets in this case.

The structure for a request object consist of the following fields in a JSON object:

  • jsonrpc: A String specifying the version of the JSON-RPC protocol. MUST be exactly “2.0”.
  • method: A String containing the name of the method to be invoked.
  • params: A Structured value that holds the parameter values to be used during the invocation of the method. This member MAY be omitted.
  • id: An identifier established by the Client that MUST contain a String, Number, or NULL value if included. this id is used in the response object to match the request.

The structure for a response object consist of the following fields in a JSON object:

  • jsonrpc: A String specifying the version of the JSON-RPC protocol. MUST be exactly “2.0”.
  • result: This member is REQUIRED on success. This member MUST NOT exist if there was an error invoking the method. The value of this member is determined by the method invoked on the Server.
  • error: This member is REQUIRED on error. This member MUST NOT exist if there was no error triggered during invocation.
  • id: This member is REQUIRED. It MUST be the same as the value of the id member in the Request Object.

Setting up the project

To start using the cortex API, some plugins are needed to ease the process of communication. The first one is the WebSocket plugin, which can be found on the asset store, Import the package so the WebSocket class becomes available.

The project window should look like this.

WebSocket plugin ready to use

Next, we need the JSONObject repository, download it as a zip and decompress it at the plugins folder, it should look like the following.

JSONObject class ready to use

Finally, take from the Cortex-Implementation-For-Unity repository, the CortexJsonUtility class and locate it on a the Plugins/Cortex folder.

CortexJsonUtility class ready to use

Connecting through the WebSocket

Now that the resources needed are present in the project, let’s create a Script called EmotivTest which inherits from MonoBehaviour and add it as a component in a GameObject called Emotiv.

Emotiv GameObject
using System;
using System.Collections;
using System.Collections.Generic;
using UnityEngine;
public class EmotivTest : MonoBehaviour
{
}

As working with WebSocket requires to wait for responses, we are going to have all of the code related to it on coroutines. First, let’s add a “Start” coroutine.


public class EmotivTest : MonoBehaviour
{
private IEnumerator Start()
{
}
}

And on this coroutine, we’re going to create an Instance of WebSocket class. The WebSocket constructor ask for an instance of the Uri class, and on this instance of the Uri we’re passing the emotiv url as the parameter, that url will always be : “wss://emotivcortex.com:54321”.

Finally, we’re going to tell the program to wait until the websocket finishes connecting


private IEnumerator Start()
{
WebSocket w = new WebSocket(new Uri(“wss://emotivcortex.com:54321”));
yield return StartCoroutine(w.Connect());
}

And now the program is ready to start communicating with the Emotiv through the WebSocket.

Emotiv’s request’s documentation

To create the request JSON objects, we’re going to use the CortexJsonUtility class, which consist of three public static methods that facilitates the process of creating the JSON-RPC objects needed for the requests to the cortex service and also to gather some information from the cortex responses.

To start a connection with the Emotiv device we need to fist login in the cortex service, for this we need to request a login method, the documentation of all methods can be found on this link https://emotiv.github.io/cortex-docs/#introduction.

The login method requires you to provide a username and password of the user from the Emotiv Cloud, and the client id and secret for the application. Those requirements are listed in a table in the documentation page as follows:

login method params

For the username and password the user has to sign in at the Emotiv webpage https://www.emotiv.com/. In the case of the client_id and client_secret, it’s necessary to create an app on the developer dashboard of the Emotiv webpage https://www.emotiv.com/my-account/cortex-apps/.

After you create the app the client_secret is revealed, take note of it as it will be the only time it will be shown. The client_id on the other hand, can be found on the cortex-apps link.

Client ID and Client Secret

Once all the parameters have been gathered, we can create the JSON-RPC object for the login method.

Login

At this point the username and the password could be sent directly, but let’s make a window with some input field and a button so the user is able to provide the username and password, then click on a button to login. Something like this:

Login canvas

Now we need to modify the EmotivTest script so we can access the WebSocket connection initiated earlier.

First, let’s move the WebSocket type variable from the Start coroutine to the beginning of the class as a static variable.

// static fields
static WebSocket w;

After this, the coroutine needs to be modified, the name should be changed so it becomes a coroutine started whenever we want and not at the beginning of the scene, also the w variable needs to be assigned here but without the definition part.

private IEnumerator Initialize()
{
w = new WebSocket(new Uri(“wss://emotivcortex.com:54321”));
yield return StartCoroutine(w.Connect());
}

Let’s also modified the literal parameter in the Uri constructor and replace it with a constant.

const string EMOTIV_URL = “wss://emotivcortex.com:54321”;
w = new WebSocket(new Uri(EMOTIV_URL));

And also other constants could be added to hold the client id and secret.

const string CLIENT_ID = “xxx”;
const string CLIENT_SECRET = “xxx”;

We currently need a public method to start the coroutine when the login button is pressed, so we’re going to define it.

public void Init ()
{
StartCoroutine(Initialize());
}

To finally get user’s info we need 2 static variables and 2 set methods that will work along with the input fields’ OnEndEdit event.

static string username = “XXX”;
static string userPassword = “XXX!”;
public void set_Username(string s)
{
username = s;
}
public void set_UserPassword(string s)
{
userPassword = s;
}

The code should look like the following code, but remember to change the strings in the client_id and client_secret with your own app info.

using System;
using System.Collections;
using System.Collections.Generic;
using UnityEngine;
public class EmotivTest : MonoBehaviour
{
// Constants
    const string EMOTIV_URL = “wss://emotivcortex.com:54321”;
const string CLIENT_ID = “XXX”;
const string CLIENT_SECRET = “XXX”;
    // static fields
    static WebSocket w;
static string username = “XXX”;
static string userPassword = “XXX”;
    public void set_Username(string s)
{
username = s;
}
    public void set_UserPassword(string s)
{
userPassword = s;
}
    public void Init ()
{
StartCoroutine(Initialize());
}
    private IEnumerator Initialize()
{
w = new WebSocket(new Uri(EMOTIV_URL));
yield return StartCoroutine(w.Connect());
}
}

To set up the event on the input field we drag the Emotiv GameObject to the event field, and then select the EmotivTest class and finally the UserName or UserPassword in the dynamic string group.

OnEndEdit event setup

Creating the login JSON-RPC Object

It’s time to build the request using the CortexJsonUtility class. To start, a dictionary has to be created, to hold each of the parameters needed for the login class. Add the following line, right after the last line inside the Initialize coroutine.


|Dictionary<string, string> loginDictionary = new Dictionary<string, string>();

Now the args should be added.

loginDictionary.Add(“username”, username);
loginDictionary.Add(“password”, userPassword);
loginDictionary.Add(“client_id”, CLIENT_ID);
loginDictionary.Add(“client_secret”, CLIENT_SECRET);

The final step is a line of code which creates the JSONObject and turns it to a string to then be sent to the websocket.

w.SendString(
CortexJsonUtility.GetMethodJSON
(
“login”,
1,
loginDictionary
));

The GetMethodJSON function, takes 3 arguments, a methodName which is the Method to be called (the login method this time), the methodId for the response and a dictionary with all the parameters for the method to be called.

The message has been sent but, can we know if it does receive it? Well actually we are able to do that by creating another coroutine, have a look at it.

IEnumerator GetReply()
{
while (true)
{
string reply = w.RecvString();
if (reply != null)
{
Debug.Log(reply);
}
yield return 0;
}
}

It is an infinite loop that receives responses from the websocket as a string, if the response is not null then it is printed on the console and the coroutine waits for the next frame.

To get the response from the login request let’s start the coroutine before sending the string to the websocket

private IEnumerator Initialize()
{
w = new WebSocket(new Uri(EMOTIV_URL));
yield return StartCoroutine(w.Connect());
StartCoroutine(GetReply());

Dictionary<string, string> loginDictionary = new Dictionary<string, string>();

loginDictionary.Add(“username”, username);
loginDictionary.Add(“password”, userPassword);
loginDictionary.Add(“client_id”, CLIENT_ID);
loginDictionary.Add(“client_secret”, CLIENT_SECRET);

w.SendString(
CortexJsonUtility.GetMethodJSON
(
“login”,
1,
loginDictionary
));
}

To try this, assign the Init function to the login button, enter a proper username and password, also make sure the client id and secret are correct, after pushing the button then you should get on the console 1 message similar to this:

{“id”:1,”jsonrpc”:”2.0",”result”:”User xxx login successfully”}

The request is also a string in a JSON-RPC format and if it has a field named “result” then the request was a success, or maybe something went wrong and an error was shown.

{“error”:{“code”:-32001,”message”:”Incorrect user name or password.”},”id”:1,”jsonrpc”:”2.0"}

This error is shown if any of the parameters is wrong, including the client id or secret. Or maybe the error was.

{“error”:{“code”:-32032,”message”:”Logout before login with new account.”},”id”:1,”jsonrpc”:”2.0"}

Which is shown if a user is already logged in, if that’s so, logout manually using the cortex UI or send a logout request to the cortex following the same process of the login above.

Handling responses

It is possible to react to specific responses from the cortex due to the id given, the response message will always match the id of the request, in this case we could hide the login panel as we don’t want the user to re-login, instead we could show another panel with the logged user info and a logout button.

User info canvas

The next steps aren’t the appropriate way to do things, but it will work for testing purposes.

We’re adding 2 private variables to hold our canvas gameobjects and be able to activate and deactivate them, this variables will have the [SerializeField] field attribute so we can add them on the inspector.


// private fields
[SerializeField]
GameObject loginCanvas;
[SerializeField]
GameObject userInfoCanvas;

We are also going to need an enumeration outside of the class so the method’s id is given in an ordered way, this will have Login and Logout for now, but everytime you implement a method, it should be added in this enumeration.

public enum MethodsID
{
Login,
Logout
}

And now let’s replace the login method call, changing the 1 with (int)MethodsID.Login, so it should look like this.

w.SendString(
CortexJsonUtility.GetMethodJSON
(
“login”,
(int)MethodsID.Login,
loginDictionary
));

This is the moment to get info from the response string, first we have to convert the string into a JSONObject, then we are able to get each field of the response, the one we need is a field called “result” and if this field exist we’re able to get the id from it. let’s add some code to the GetReply method we already have, just inside the if statement and after the debug.


JSONObject replyObj = new JSONObject(reply);
// If there's a field named “result”, it is the actual response to the sent method
JSONObject resultObject = replyObj.GetField(“result”);
if (resultObject)
{
JSONObject idObject = replyObj.GetField(“id”);
}

and now we can create a switch to handle the code depending on the id.


if (resultObject)
{
JSONObject idObject = replyObj.GetField(“id”);

switch ((MethodsID)(int)idObject.n)
{
case MethodsID.Login:
// Handle Login Success
break;
case MethodsID.Logout:
// Handle Logout Success
break;
}
}

As we are going to use the Unity UI we need to add this library to the EmotivTest script

using UnityEngine.UI;

To handle the Login response we will need a function that deactivates the login canvas and activates the user info canvas, then it will show the username variable content in the text field inside the canvas.

void HandleLoginSuccess()
{
loginCanvas.SetActive(false);
userInfoCanvas.SetActive(true);
userInfoCanvas.transform.GetChild(0).GetChild(0).GetComponent<Text>().text = username;
}

Finally it should be called on the MethodsID.Login case.

switch ((MethodsID)(int)idObject.n)
{
case MethodsID.Login:
HandleLoginSuccess();
break;

Logout

We are adding 2 functions, one to send the logout method request and the other to hide the user info canvas and show the login one.

public void Logout()
{
Dictionary<string, string> logoutDictionary = new Dictionary<string, string>();
    logoutDictionary.Add(“username”, username);
w.SendString(
CortexJsonUtility.GetMethodJSON
(
“logout”,
(int)MethodsID.Logout,
logoutDictionary
));
}
private void HandleLogoutSuccess()
{
loginCanvas.SetActive(true);
userInfoCanvas.SetActive(false);
}

To set this up the canvas needs to be assigned to the EmotivTest variables and the Logout button event needs to point to the Logout function.

Canvas assigned to the EmotivTest component
EmotivTest.Logout assigned to the OnClick event in the logout button

Test the game and now you are able to login, see the username and then logout as many times as you want.

The EmotivTest class should look like this:

using System;
using System.Collections;
using System.Collections.Generic;
using UnityEngine;
using UnityEngine.UI;
public enum MethodsID
{
Login,
Logout,
}
public class EmotivTest : MonoBehaviour
{
// Constants
const string EMOTIV_URL = “wss://emotivcortex.com:54321”;
const string CLIENT_ID = “xxx”;
const string CLIENT_SECRET = “xxx”;
    // static fields
static WebSocket w;
static string username = “XXX”;
static string userPassword = “XXX!”;
    public void set_Username(string s)
{
username = s;
}
    public void set_UserPassword(string s)
{
userPassword = s;
}
    // private fields
    [SerializeField]
GameObject loginCanvas;
    [SerializeField]
GameObject userInfoCanvas;
    public void Init ()
{
StartCoroutine(Initialize());
}
    private IEnumerator Initialize()
{
w = new WebSocket(new Uri(EMOTIV_URL));
yield return StartCoroutine(w.Connect());
StartCoroutine(GetReply());
        Dictionary<string, string> loginDictionary = new Dictionary<string, string>();
        loginDictionary.Add(“username”, username);
loginDictionary.Add(“password”, userPassword);
loginDictionary.Add(“client_id”, CLIENT_ID);
loginDictionary.Add(“client_secret”, CLIENT_SECRET);
        w.SendString(
CortexJsonUtility.GetMethodJSON
(
“login”,
(int)MethodsID.Login,
loginDictionary
));
}
    IEnumerator GetReply()
{
while (true)
{
string reply = w.RecvString();
if (reply != null)
{
Debug.Log(reply);
JSONObject replyObj = new JSONObject(reply);
// If there's a field named “result”, it is the actual response to the sent method
JSONObject resultObject = replyObj.GetField(“result”);
if (resultObject)
{
JSONObject idObject = replyObj.GetField(“id”);
switch ((MethodsID)(int)idObject.n)
{
case MethodsID.Login:
HandleLoginSuccess();
break;
case MethodsID.Logout:
HandleLogoutSuccess();
break;
}
}
}
yield return 0;
}
}
    void HandleLoginSuccess()
{
loginCanvas.SetActive(false);
userInfoCanvas.SetActive(true);
userInfoCanvas.transform.GetChild(0)
.GetChild(0).GetComponent<Text>().text = username;
}
    public void Logout()
{
Dictionary<string, string> logoutDictionary = new Dictionary<string, string>();
        logoutDictionary.Add(“username”, username);
        w.SendString(
CortexJsonUtility.GetMethodJSON
(
“logout”,
(int)MethodsID.Logout,
logoutDictionary
));
}
    private void HandleLogoutSuccess()
{
loginCanvas.SetActive(true);
userInfoCanvas.SetActive(false);
}
}

Follow the process in the next article Part 2