Web AR with smart plugs using A-Frame
Noun: A technology that superimposes a computer-generated image on a user’s view of the real world, thus providing a composite view.
Last year, myself and a few other developers from The Unit went to The Full Stack Fest in Barcelona. Whilst there we were inspired by many of the talks, including Shagufta Gurmukhdas’ presentation on Web Based Virtual Reality. Afterwards we sat on our balcony with a few Estrella beers and played with A-Frame. We challenged one of the developers to a micro-hack: to build out a 3D model in 10 minutes. This just proved how quick and easy it was to use.
On our return from Barcelona we had two Developer Days for innovation. Developers had two days off from client work to focus on something new and innovative. In order to get the best out of the two days we thought it would be more effective to come up with a theme to constrain ourselves to a specific technology. Off the back of Full Stack Fest, we decided to focus on Augmented Reality (AR) and Virtual Reality (VR) — A-Frame in particular. After brainstorming a number of ideas we narrowed them down; each of us chose what we wanted to work on and started to build out each of our individual projects.
After the two innovation days we were given three weeks to further a couple of the ideas. This blog post will describe one of the ideas we had and how it progressed over the following weeks.
The idea : enARgy
One of the ideas that excited us the most was the possibility to visualise the real time energy usage of different appliances through AR. 🔌⚡️
We thought it would be cool if you could point your phone’s camera at an appliance and see the amount of energy it was currently using, overlayed with augmented reality. This was a perfect project to play around with augmented reality and A-Frame.
Some examples of augmented reality apps include:
- Pokemon GO
- Ikea Place — Ikea released an app where you can experiment with different pieces of furniture in your home, to see if it goes well in that space. You can place multiple objects and capture how a room looks with these new items.
- Google Translate — if you hold your phone over a word or sentence then it will translate it in place of the original.
AR enhances a user’s experience in the ‘real world’, whether that’s through the things that we see, hear or feel.
There are different types of AR:
- Marker-based AR (I will go into more detail about this particular type of AR a bit later on)
- Projection-based AR
- Markerless AR
- Superimposition-based AR
A-Frame
A-Frame is a WebVR framework that Mozilla released just over a year ago. A-Frame is an open source web framework with open standards and an open culture, made easy for building out WebVR apps.
A-Frame with JavaScript and canvas
We can use JavaScript and DOM APIs to control the entities within A-Frame as we would in normal web development: this is one of the many benefits of it being a HTML framework. This also means we can use all of the popular libraries such as D3, Vue.js, React, Redux, jQuery and Angular.
Backend
We are using node.js and a number of packages to retrieve the data from the smart plug. There were certain packages that we used to get the backend up and running.
We already had some WeMo smart plugs in the office, so we created a simple express web application and used wemo-client to discover devices on a local network, retrieve power consumption data and listen for events such as changes in power consumption or the status of the smart plug. We then added Socket.io to our server, which allowed us to broadcast these events over a websocket.
In order to see if any devices were connected we set logs to output in the terminal, which gave us the Mac Address of the device connected, and we could also see what port was being used to connect.
We wanted our devices to be normal, household appliances that each showed different variants of energy. Initially we just started with one appliance.
I will go through some of the code in order to help you understand how I created the backend. I will explain how you can do this using the WeMo smart plugs as we did, but if you don’t have the smart plugs then you could just fake the data.
In the code below we created our express app and attempted to connect to any smart plugs that had been discovered by the WeMo-client library.
ngrok
We had to run the app through a secure domain because you can only access the camera needed to detect the marker if the site is served from a secure domain. Using ngrok allows you to expose your local environment over a secure connection.
Frontend
We sketched out some initial ideas of how we could display useful data that we were retrieving from the smart plugs. We went through a couple of different iterations of this, which I will go into in more detail in upcoming blog posts.
JavaScript
We now need to listen for information from the smart plugs on the front end. Socket.io makes this easy: we simply need to connect to our socket server over ngrok, and then listen for the ‘instantPower’ event that we emit after receiving data from the smart plug.
We then store the last 10 pieces of data in an array and call a ‘drawGraph’ function that will update our page, making use of the new data.
Now that we have the data coming through we can start to do some cool stuff in the frontend, which I will go through in my next blog post.
Installation of A-Frame
It only takes a few minutes to add all the code required to create a scene and add your first 3D objects.
Setting up was really easy to do:
- Add the scene
- Add the camera
- Add the different entities — sphere / cylinder etc.
The HTML-based markup language has Three.js under-the-hood, enabling developers to create 3D scenes using only HTML, and making it accessible for almost any developer. The framework allows developers to create fantastic, cross-platform AR/VR experiences on the Web for anyone. Three.js can be intimidating for new developers. There are a lot of gates to get through when building for AR/VR using Three.js. Developers have to set up a camera, initialise the scene, set up the lights, declare and pass canvas element, import WebVR polyfil, install VREFFECT, instantiate renderer, create render loop, preload assets, figure out how to handle responsiveness, and deal with meta tags. On mobile, AR/VR apps can often be inconvenient for the end user; a lot of the time they need to download apps or other things to make them work. A-Frame makes all of these problems simpler to solve.
Within just 15 minutes it’s possible to add entities to a scene, style them and animate them. It’s perfect for a developer who wants to concentrate on having an easy to use playground and get results quickly.
The out of the box functionality that A-Frame offers includes camera, geometry, material, light and sound. It comes ready for stereo viewing in VR headsets and with WASD / arrow movement and camera panning / rotation support. We aren’t utilising those attributes of A-Frame in this particular project, but it’s an example of how easy it is to use and set up.
Naturally, due to the simplicity of A-Frame, it does mean that developers will eventually hit a wall with what they can achieve. On the other hand, there are some pretty complex projects out there and it doesn’t take away from how valuable A-Frame is. A-Frame has a lot of potential and has already proven itself to be extendable. One example is Jerome Etienne’s library ar.js — I will go into more detail about this shortly. A-Frame has demonstrated itself to be hugely popular (and growing), with a great, thriving community.
A-Frame inspector
A-Frame has a built-in inspector which you can enable by:
This inspector is extremely useful; when first using A-Frame the positioning, rotation and radius seemed a little tricky and confusing.
Opening up inspector in the browser makes it a lot easier and way more efficient to move entities. You can physically move each of the entities around, and when doing so you can see the values for the X, Y and Z for the position and rotation. When each of the entities are in the correct place, you can copy the numerical values and put them in the code. You can also change the scale.
A-Frame with AR
As I briefly mentioned earlier @jerome_etienne (the author of Learning Three.js Blog), wrote a library — ar.js in order to get AR running 100% in the web browser, which means no app needs to be installed and it’s supported across multiple platforms and devices running at up to 60 frames per second.
It’s a great library and has made it really easy for us to be able to achieve what we wanted to do. It worked out of the box with the standard ‘Hiro’ marker which was amazing! In the next blog post I will show what we had to do to get custom markers working. So a huge thanks to Jerome for this awesome library! 🤓
Markers
As I mentioned earlier, ‘marker-based’ is a type of AR. You have to point a camera at a marker in order to trigger a reaction. In this case we are pointing the camera at a marker and the energy usage is coming back. In this specific example we have used D3 to show the data coming back, but I won’t be going into too much detail about that.
Just for reference, here’s the complete code of our first iteration of this project. Please don’t forget that this won’t work out of the box for you, as you may not have smart plugs and we hard-coded a specific ngrok link.
In the next blog post I will discuss in greater detail how we managed to visualise energy using A-Frame and canvas in a cool way. Feel free to contact me in the meantime via Twitter @lilGemVincent. 👋🏽
Originally published at gist.github.com.
