IOT+XR
Contactless experience using IoT XR
Concept implementation of controlling devices without touching them using IoT and XR.
Covid19 is forcing us to adopt new ways of working, see the below article by Kuldeep Singh and Hariharasudhan R, and see how human behaviors are changing, and the virtual is becoming new real. The need for touchless or contactless experience is increasing.
In this article, I am sharing my learnings while implementing an app named “ContactLess”. We are going to press the switch button that controls an LED, from a distance, we will be detecting the device and the switch button using image processing and, create a virtual button on the top physical button, that we can control in XR.
Before we start
We will use MQTT and Vuforia Image detection techniques. Those who wonder what is MQTT here is a small article on MQTT
And also here is an article on image target and detection using Vuforia, here is an interesting article by Jae Salavarrieta
Next, we need is an MQTT broker, I am using a mosquito broker in my local network. You will also need an MQTT DLL file to use MQTT functionalities, you can use this link to download it here. If you are new to Unity, please go through below article from Neelarghya
Lets Start
Let’s get started in the world of XR and IoT.
Setup Image Target
Create your image target and put inside the ARCamera of Unity, you can get it from GameObject -> VuforiaEngine -> ARCamera
MQTT Setup in Unity
Add MQTT dll inside the assets.
Now let’s try to connect to MQTT broker and the network
Now we will add a script to the image target, select the ImageTarget in the hierarchy on the inspector window click Add Component button -> New Script -> name your script I named it as MqttConnect.
To connect with the broker we need broker Host address, broker port, and Wi-Fi details. I am having a local mosquito broker in my pc, so I can use my IP and for port address, most commonly 1883 will be used.
So we will connect to the network and MQTT broker on the Start lifecycle.
public class MqttConnect : MonoBehaviour
{
private MqttClient client;
private string brokerHostname = "your MQTT broker address";
private int brokerPort = "MQTT broker address port";
private string userName = "your Wi-Fi user name";
private string password = "your Wi-Fi password";void Start()
{
if (brokerHostname != null && userName != null && password != null)
{
Connect();
}
}private void Connect()
{
client = new MqttClient(brokerHostname);
string clientId = Guid.NewGuid().ToString();
try
{
client.Connect(clientId, userName, password);
}
catch (Exception e)
{
Debug.LogError("Connection error: " + e);
}
}
}
Create a Virtual Switch Button
You can create a normal UI button by right click -> UI -> button, I am adding it under Image target, since it will create only when the target is detected and naming it as on/off button.
For the virtual button, I am going to use the button image(jpg format) and convert it to the sprite and apply to the button image sprite. Click on the image and see the inspector window and in the texture, type change it to the sprite 2D and UI option. Similarly, do for both on and off images.
Now add the off sprite to the button
Now we will create an MQTT publish functionality :
void Publish(string msg)
{
client.Publish(
"esp-led", Encoding.UTF8.GetBytes(msg), MqttMsgBase.QOS_LEVEL_EXACTLY_ONCE, // QoS level
false // retained);
}On this method, the first parameter is the topic and second is the message to be transmitted in UTF8.
Now we will create the toggle functionality when the button is clicked:
[SerializeField] private Button onOffButton;
[SerializeField] private Sprite onSprite;
[SerializeField] private Sprite offSprite;
private bool toggleValue = true;So we will create an instance of a button, on sprite, off sprite and boolean value and then we will add an on-click listener for the button in the Start method. Sprites need to change the transition when the button is clicked.
onOffButton.onClick.AddListener(ChangeState);// in start methodvoid ChangeState()
{
if (toggleVal)
{
onOffButton.image.sprite = offSprite;
Publish("off");
toggleVal = false;
}
else
{
onOffButton.image.sprite = onSprite;
Publish("on");
toggleVal = true;
}
}
At last, you can add the on/off button in the inspector for MqttConnect script
That’s it from the unity side you can run the app
IoT Device Setup
To verify this I had tested with Esp8266 and a led, which you saw on the top
And also I will add the code for Esp8266, to connect with Wi-Fi and the MQTT broker.
#include <ESP8266WiFi.h>
#include <PubSubClient.h>const char* ssid = "Wi-Fi id";
const char* password = "Wi-Fi password";
const char* mqttServer = "Mqtt broker address or Ip";
const int mqttPort = 1883;//broker port numberWiFiClient espClient;
PubSubClient client(espClient);void setupWifi() {
Serial.println();
Serial.print("Connecting to ");
Serial.println(ssid);
WiFi.begin(ssid, password);int tries = 0;
while (WiFi.status() != WL_CONNECTED) {
delay(500);
Serial.print(".");
tries++;
if (tries>20) {
Serial.print("Connection failed. Rebooting...");
ESP.restart();
}
}Serial.println();
Serial.println("WiFi connected");
client.setCallback(callback);
Serial.println("IP address: ");
Serial.println(WiFi.localIP());
}void reconnect()
{
while(!client.connected()){
Serial.println("Attempting MQTT connection");
if(client.connect("client")){
client.subscribe("esp-led");
Serial.println("MQTT connected");
}
else {
Serial.print("failed, rc=");
Serial.print(client.state());
Serial.println(" trying again in 2 seconds");
delay(2000);
}
}
}void setup() {
Serial.begin(115200);
pinMode(5, OUTPUT);
setupWifi();
client.setServer(mqttServer, mqttPort);
}void callback(char* topic, byte* payload, unsigned int length) {
Serial.print("Message arrived in topic: ");
Serial.println(topic);
String str = "";
Serial.print("Message:");
for (int i = 0; i < length; i++) {
Serial.print((char)payload[i]);
str += (char)payload[i];
}if(str == "on"){
digitalWrite(5, HIGH);
}
else{
digitalWrite(5, LOW);
}
Serial.println();
Serial.println("-----------------------");
}void loop() {
if (!client.connected()) {
Serial.println("MQTT not connected");
reconnect();
}
client.loop();
}
Demo
Just Open the Contactless App and enjoy!
Credit for button image goes to Aranjuezmedina / Freepik
Conclusion
We have seen a basic working of contactless experience, we can extend this use case to a wider scale of integrating it with other IoT devices, just see a controllable device from the lens of your phone and control it.
Keep in touch, will share my more experiments soon…
