“Alya-Smart-Mirror” OpenIoTChallenge Final Report
There are already many impressive open-source smart-mirror projects out there but we wanted to achieve even more. We were inspired by the MagicMirror project which is a modular smart-mirror with support for a wide range of 3rd-party modules that adds more features to the mirror such as showing the weather, reading the news, or controlling lights at home. “alya-smart-mirror” is another DIY open-source smart mirror that has voice/visual/gestures recognition at its core. The idea is to build a completely personalized smart mirror that configure itself automatically based on the user who is standing in-front of it. “alya-smart-mirror” is built in a flexible way to be modular and supports 3rd-party add-ons. We want also to go even more by supporting an easy to use hybrid app that can be used by the user to completely configure the mirror and all the add-ons.
Our overall goals are:
- Completely open-source with good documentations to explain all the steps needed to build and configure the mirror.
- Support for different visual/voice/gestures recognition options to make it more flexible for users based on the requirements with focus on the offline options to make it affordable.
- Modular and provides an easy way for contributors to develop, test and integrate the modules with “alya” core. All 3rd-party add-ons/modules should receives alerts about the user standing in-front of the mirror allowing them to develop personalized features. Similarly the add-ons can subscribe to optional gesture/sensor events.
- Provide good user experience by having a better accessibility, usability, and providing more efficient way for the daily interactions between the user and the mirror. For that we aim to provide a mobile app that the user can use to control and configure all the aspects of the mirror.
An overview of the components in “alya-smart-mirror” are shown below:
- Mirror: is the main component which has all the hardware
- Backend: a server that provides all the APIs required by the mirror app/mobile app. The backend also interact with external cloud services such as amazon web services. The backend can be deployed either in the mirror or in the cloud. (more details: “asm-backend” the backend for alya smart mirror)
- The mobile app: the user mobile app that will be used to trigger the face recognition training process and to configure all aspects of the mirror. (more details : “asm-app” a react native mobile app to configure alya-smart-mirror)
We were lucky to be one of the 12 teams who won the starter kits. Therefore we had the chance to build something cool. We also received the UP squared IoT development kit as a gift from intel which gave us the possibility to do much more. An overview of our hardware stack is shown below:
Alya-Smart-Mirror hardware overview
- UP squared board: is the heart of the project where we run the mirror app/backend. The board is so powerful and makes perfect sense for our project since at the heart of “alya-smart-mirror”, we need to do intensive processing required for the offline voice/visual/gestures recognitions. We are also using xbox kinect 2 which can be connected only to limited boards since it requires USB 3 and faster CPUs.
- xbox Kinect 2: used for advance gestures/body tracking.
- Matrix Creator one: although in theory having the above is enough for our use cases (voice/visual/gesture recognition), we wanted to test the matrix creator which abstract the complexity of writing voice/visual recognition applications using their matrix OS. It also has a built-in sensors and support for zigbee/zwave protocols which makes it the perfect hardware for smart home applications. We are using it mainly for voice/face recognition, collecting sensor data, and controlling home appliances.
- Raspberry pi 3 model B: unfortunately we couldn’t make the “matrix creator one” works with the UP squared board since it is not compatible. Using the raspberry pi was the only option.
- Typical smart mirror hardware: this includes the monitor, two-way mirror and a first aid cabinet.
All the details about how designed the hardware part can be found in this blog.
An overview of the software stack is show below:
Alya-Smart-Mirror software overview
- Alya core: currently it is a react app that runs over electron as a desktop application. The core has the main functionality including voice/visual/gestures recognition.
- Alya component: this is an electron react component. It can be either a core or 3rd-party component.
- IoT device platform: we are using amazon IoT but any other platform can be used such as eclipse IoT, watson IoT, etc..
Open Technology and Standards
- Alexa: Alexa has a set of great built in commands as well as its openness and flexibility to build customized skills. Above that, Alexa is known of its openness and cross-platform characteristic.
- MongoDB: It is an open-source cross-platform document-based database, it is classified as a NoSQL database, it uses JSON-like documents with schemas. In Alya, MongoDB is running in Raspberry board.
- Socket.IO : We are using socket.io for sending notifications between different Alya components. In our project, we have three main components: the mirror, the mobile app, and the backend. Using sockets in general for the communication among them was a wise choice.
- MQTT Protocol: MQTT usually used for publishing and subscribing for topics. In our projects there are many sensors sending data and the addons want to get this data. Using AWS IoT MQTT was ideal for our case.
- React-Camera: The camera component we used is a ready-to-use react component called react-camera and it fits well in our use case, thanks to our friends in the open-source community who built it.
- Matrix OS : A set of libraries/ capabilities built over Matrix Creator powerful hardware.
- Pocketsphinx : PocketSphinx is a lightweight speech recognition engine, specifically tuned for handheld and mobile devices, though it works equally well on the desktop.
Applicability of a solution to a specific industry
- Smart Home : Alya is integrated with many things connected to the smart home industry. Alya can control your smart lights, and it can be developed to control more smart devices.
- Sport : Integrating Alya with Xbox Kinect gives us wide range of motion capabilities. Alya can help you in doing exercises and it can teach you new ones. Alya can be even your therapy to help you doing some needed medical exercises.
- Shops: Shop owners can build Alya Add-on to help customers trying new clothes, shoes or even glasses, this will give the users an awesome experience !
- Marketing : The capability of Alya of showing videos can be a great idea for brand marketing. Alya can be placed in shops, homes, or public places to add great marketing features for different brands.
- Gaming : Alya can play simple games with the user using voice commands. It will be able to play complex games with the users using the gesture when it gets fully integrated with the kinect.
- Entertainment : Alya is integrated with many entertaining services like Alexa, which give Alya a wide range of fun capabilities like playing songs, joking, conversations and many great things.
- Workplace : Alya is using offline face recognition service, which can detect , for example, the employers in and out the company and if there is stranger came into the company. Simple addon can be added to check the employers and calculate the statistics of their attendance or fire the alarm if there is a stranger in a wrong place.
- We learned more about MQTT messaging, the scenarios where it can be useful and its pros and cons.
- We gained more knowledge about socket-based communication, how we can use it to send notifications, best use-cases where sockets can fit well and its pros and cons.
- Document-Based database and specifically MongoDB, how it is efficient when dealing with key-value based data.
- Alexa and how we could use Alexa in Raspberry Pi, and the good experience of creating customized skills for Alexa.
- React Native: how we can customise mobile application per single user and how to link it with real data communication using client io sockets and APIs
- Redux: how to store data needed by multiple screens of your app in one place and enable them to communicate smoothly.
- Online Visual Recognition, how to use cloud-based solutions to build a face-recognition service.
- Designing a framework that can be extended to 3rd parties can be a challenge. Using reactjs with electron was interesting and we had few problems that had to fix. Our goal was to give the developers a flexible options and at the same time should be easy to start developing an add-on for alya-smart-mirror. We are going to create a boilerplate for developers and investigate the use of other frameworks such as emberjs.
- Not all boards can handle complex processing needed for gestures/visual recognitions. Also when it comes to connecting to a powerful hardware such as xbox kinect, a strong board is needed. For that we leveraged the intel up-squared board.
- The accuracy of voice/visual/gestures recognition can be a challenge and so much dependent on the hardware. For example, having a single microphone compared to an array of microphones can have a huge difference on results. Similarly, using normal camera or a powerful one such as xbox kinect.
- IoT cloud-based brokers can be expensive if we use it for all type of events, for that we had to use socket notifications and send only important events to the cloud.
- Our focus was also the usability for the end user, for that we have created an easy to use mobile app, but we think we still need to do more in this regards. We have a list of ideas for the future such using the app to completely control the UI of the mirror.
- Support more visual/voice/gesture options
- Support more gestures using xbox kinect
- Better documentations
- Automate most of the steps to easily deploy/install alya-smart-mirror by simple end users.
- Adding support for user personalized notifications which might includes facebook & google calendar to start with.
- More add-ons to cover many use cases such as weather, traffic, maps , booking taxi, etc …
- App: We have prepared a short demo for Alya mobile Application:
- Full demo for all the parts working together:
- Blog: “alya” a DIY modular personalized open-source smart mirror
- Blog: Cloud voice recognition “Alexa” for “alya-smart-mirror”
- Blog: Offline gesture recognition for “alya-smart-mirror””
- Blog: “alya-smart-mirror” as a sensor hub for your home
- Blog: “asm-app” a react native mobile app to configure alya-smart-mirror
- Blog: Offline voice recognition for “alya-smart-mirror”
- Blog: “asm-youtube-addon” a react component for getting youtube videos using voice commands
- Blog: “asm-backend” the backend for alya smart mirror
- Blog: “alya-smart-mirror’ offline visual recognition core
- Blog: Cloud-based Face Recognition for Alya Smart Mirror
- Blog: Alya-Smart-Mirror from vision to reality