Videogame control through EEG (part 2)

This post is the continuation of a previous post: Videogame control through EEG (part 1) and following it, we will go forward in the process to get data from the Emotiv headset.

The process to get mental commands

As the cortex documentation points, we need the following process to get mental commands of a given action:

  1. Authenticate a user
  2. Create a session
  3. Subscribe to the sys event
  4. Setup training neutral action
  5. Accept or reject the training
  6. Subscribe to the com event

Authenticating a user

To receive the authorization token we need to define an Authorize method that sends the authorize method to the cortex, with the client id and client secret as parameters.

void Authorize()
{
Dictionary<string, string> authorizeDictionary = new Dictionary<string, string>();
authorizeDictionary.Add(“client_id”, CLIENT_ID);
authorizeDictionary.Add(“client_secret”, CLIENT_SECRET);
w.SendString(
CortexJsonUtility.GetMethodJSON
(
“authorize”,
(int)MethodsID.Authorize, // Add Authorize into the enum
authorizeDictionary
));
}

Add a call to this function inside the Initialize coroutine, just after sending the login method.

Authorize();

To receive the authorization token we need to create a new string variable and add another case to the GetReply switch.

string _auth;
case MethodsID.Authorize:
_auth = CortexJsonUtility.GetFieldFromJSONObject(replyObj, “_auth”);
break;

This looks for a field named “_auth” inside the result object in the reply from the cortex and assigns it to the _auth variable that will be used to create a session in the next section.

Creating a session

It’s time to create a session and tell the cortex to start managing data streams from the Emotiv.

IEnumerator CreateSession()
{
while (_auth == null)
{
yield return 0;
}

Dictionary<string, string> sessionDictionary = new Dictionary<string, string>();
sessionDictionary.Add(“_auth”, _auth);
sessionDictionary.Add(“status”, “open”);
w.SendString(
CortexJsonUtility.GetMethodJSON
(
“createSession”,
(int)MethodsID.CreateSession, // Add CreateSession into the enum
sessionDictionary
));
}

This coroutine can be called after the Authorize method inside the Initialize coroutine and sends a createSession method to the cortex right after the _auth variable is assigned with something other than a null.

At this point if we play the game and enter the login button it will only work if there’s an Emotiv device connected, on which the session will be created.

Subscribing to events

To receive data from a session, you have to subscribe to it using the subscribe method. Along with the session you want to subscribe to, you have to specify which streams you are interested in. Each stream represents a different kind of data coming from the headset, this time we’re going to subscribe to 2 of the available streams. The “sys” stream which is a system event used to know the stages of a training and the “com” which is a mental command event called each time an action is perceived by the headset after it is trained.

IEnumerator SubscribeStreams()
{
while (_auth == null)
{
yield return 0;
}
w.SendString(
CortexJsonUtility.GetSuscribtionJson
(
(int)MethodsID.Suscribe,
_auth,
new string[] { “sys”, “com” }
));
}

This coroutine can be called after the starting the CreateSession coroutine at the end of the Initialize coroutine.

Training an action

It’s time to create another canvas to train our 2 actions, neutral and right.

Training Canvas

This is the main part of our code and consist of 2 parts, an initial code to start the training of an action and a final one to accept or reject the training.

The idea is that the user clicks on the action that is going to be trained, then the user will think of that action for as long as the emotiv decide it succeeded or failed to record the action, then a message will pop up letting the user accept or reject the action and if the user accepts, the “com” events of that given action will start to pop up.

The first thing we need are 2 boolean variables to control the training process, isTrainingComplete which will avoid the call of multiple traing messages at the same time and isTrainingComplete which will make the coroutine hold the process until the user finishes the training. So we are adding them in the static fields section.

static bool isTrainingDone;
static bool isTrainingComplete = true;

As the game will show a message with 2 options to be selected after succeeding in the training and depending on the option a new training method will be called with different parameters, we should define a method just to send the message with the arguments passed.

public void SendTrainMessage(Dictionary<string, string> parameters)
{
w.SendString(
CortexJsonUtility.GetMethodJSON
(
“training”,
(int)MethodsID.Training,
parameters
));
}

Then we define a coroutine for the whole process that needs 2 string parameters, the detection type and the action to be trained.

IEnumerator InitializeTraining(string detectionType, string action)
{
}

all the code will be located inside of a if statement that will check if the isTrainingComplete is true, so no more training methods are called at the same time.

if (isTrainingComplete)
{
}

Now we create a dictionary with the arguments needed to a training method.

Dictionary<string, string> initialParams = new Dictionary<string, string>();
initialParams.Add(“_auth”, _auth);
initialParams.Add(“detection”, detectionType);
initialParams.Add(“action”, action);
initialParams.Add(“status”, “start”);

After it, we call the method created earlier, set the control variables to false and pause the process until the training is done.

SendTrainMessage(initialParams);
isTrainingDone = false;
isTrainingComplete = false;
while (!isTrainingDone)
yield return 0;

To know when the training succeeded or failed the Cortex send the “sys” events that can be handled on the GetReply function.

JSONObject sysArray = replyObj.GetField(“sys”);
if (sysArray)
{
switch (sysArray[1].str)
{
case “MC_Succeeded”:
isTrainingDone = true;
break;
case "MC_Completed":
isTrainingComplete = true;
break;
case "MC_Rejected":
isTrainingComplete = true;
break;
case "MC_Failed":
isTrainingComplete = true;
break;
}
}

Before retaking the InitializeTraining coroutine, we will need a class that handles the message shown to the user that will let choose to reject or accept the training, create a new script and add the following code.

using System.Collections;
using System.Collections.Generic;
using UnityEngine;
using UnityEngine.UI;
public class MessageQuestionSelection
{
/// <summary>
/// Shows a menu with a given text and lets the user press one of two buttons.
/// These buttons are assigned to a certain delegated passed as
parameter. Finally, this functions on the delegates are called using the aproppiate arguments passed as parameter.
/// </summary>
/// <param name=”messageQuestion”></param>
/// Provided text to be shown.
/// <param name=”answer”></param>
/// delegate containing the function that will be executed once the button is pressed.
/// <param name=”acceptParams”></param>
/// String array with the accept parameters needed to execute the method in the delegate.
/// <param name=”rejectParams”></param>
/// String array with the reject parameters needed to execute the method in the delegate.
public static void ShowMessageWithSelection(string messageQuestion, DelegateSelectionMenu answer, Dictionary<string, string> acceptParams, Dictionary<string, string> rejectParams)
{
GameObject messageWithSelection = GameObject.Find(“Canvas_Selection_Message”);
if(messageWithSelection == null)
{
messageWithSelection =
GameObject.Instantiate(Resources.Load(“Canvas_Selection_Message”, typeof(GameObject))) as GameObject;
messageWithSelection.name = “Canvas_Selection_Message”;
}
messageWithSelection.SetActive(true);
messageWithSelection.transform.GetChild(0)
.GetChild(0).GetComponent<Text>().text = messageQuestion;
messageWithSelection.transform.GetChild(0)
.GetChild(1).GetComponent<Button>().onClick.AddListener(() => answer.Invoke(acceptParams));
messageWithSelection.transform.GetChild(0)
.GetChild(2).GetComponent<Button>().onClick.AddListener(() => answer.Invoke(rejectParams));
}
}
/// <summary>
/// A delegate is created to be used for the ShowMessageWithSelection answer method.
/// </summary>
/// <param name=”parameters”>
/// Call ShowMessageWithSelection answer.
/// </param>
public delegate void DelegateSelectionMenu(Dictionary<string,string> parameters);

This class to work needs a prefab to be constructed and saved in a folder called Resources and named Canvas_Selection_Message, it should have the following children.

Canvas_Selection_Message prefab structure

It also defines a delegate to assign the SendTrainMessage method to each button and that is what we are going to do now, at the end of the InitializeTraining coroutine create an instance of the DelegateSelectionMenu and assign the method.

DelegateSelectionMenu delegateSelection = SendTrainMessage;

The reject and accept parameters need to be build so we can call the ShowMessageWithSelection function in the MessageQuestionSelection class.

Dictionary<string, string> acceptParams = new Dictionary<string, string>();
acceptParams.Add(“_auth”, _auth);
acceptParams.Add(“detection”, detectionType);
acceptParams.Add(“action”, action);
acceptParams.Add(“status”, “accept”);
Dictionary<string, string> rejectParams = new Dictionary<string, string>();
rejectParams.Add(“_auth”, _auth);
rejectParams.Add(“detection”, detectionType);
rejectParams.Add(“action”, action);
rejectParams.Add(“status”, “reject”);
MessageQuestionSelection.ShowMessageWithSelection
(
“Do you accept the training of “ + action.ToUpper() + “?”,
delegateSelection,
acceptParams,
rejectParams
);

And that will do the training process, now we need a public function to start the coroutine when the button is pressed.

public void Train(string action)
{
StartCoroutine(InitializeTraining(“mentalCommand”, action));
}

Finally the train panel need to be shown after the session is successfully created, so add to the EmotivTest class the following variable and assign the canvas to it on the inspector.

[SerializeField]
GameObject trainingCanvas;

Before starting the scene just the login canvas should be active.

On the GetReply swith add a case for MethodsID:CreateSession and there the canvas could be activated.

case MethodsID.CreateSession:
trainingCanvas.SetActive(true);
break;

Also remember to hide the panel when the user logs out, add that code in the HandleLogoutSuccess function.

private void HandleLogoutSuccess()
{
loginCanvas.SetActive(true);
userInfoCanvas.SetActive(false);
trainingCanvas.SetActive(false);
}

Moving an object on an action

We have almost done the whole process, until this point, the cortex will send a message every time it registers an action already trained, something like this.

{“com”:[“push”,0.673717498779297],”sid”:”46d18597–7034–40ab-9d6e-d617a89a24ce”,”time”:245.356536865234}

So what’s left is to tell the movable object to move every time that action is registered.

An event will work great here, add it to the EmotivTest class.

// Events
public delegate void MentalCommandEvent(string action);
public static event MentalCommandEvent OnMentalCommandEvent;

And call it when a “com” event has been received in the GetReply function, add the following lines after the sys lines.

JSONObject comArray = replyObj.GetField(“com”);
if (comArray)
{
if (OnMentalCommandEvent != null)
{
OnMentalCommandEvent(comArray[0].str);
}
}

Now create a Player class and assign it to a cube or whatever you want to move as a component.

using System.Collections;
using System.Collections.Generic;
using UnityEngine;
public class Player : MonoBehaviour 
{
private void OnEnable()
{
EmotivTest.OnMentalCommandEvent += Move;
}
private void OnDisable()
{
EmotivTest.OnMentalCommandEvent -= Move;
}
void Move(string direction)
{
if (direction == “right”)
transform.Translate(Vector3.right * Time.deltaTime);
else if(transform.position.x > 0)
transform.Translate(Vector3.left * Time.deltaTime);
}
}

On it you assign a function that receives the mental command as a string, in this case it’s the word “right”, to the EmotivTest event just created. this function moves the object to the right if the action received is equal to “right”, if that’s not the case, the object will start moving back to the initial position.

And this is the moment of truth, connect the Emotiv Headset, run the game, login, train the neutral and right action at least 3 times, then move the cube through the scene.

If you want to look at the final version of the code, it is available on the Cortex/Example/Scripts folder in the Cortex-Implementation-For-Unity repository.