Machine Learning in Smart Homes
From Smart to Helpful
As smart homes devices are becoming more connected due to IoT. Smart home devices range from lights to TVs, front door cameras and dishwashers. For the purpose of economising usage and improving power efficiency, there is measuring and logging for activities and power consumption. The body of data generated from these logs leaves much space for improvement as this data can be analysed and learned from. The purpose of integrating machine learning into smart homes is to make the home more helpful and responsive to users needs and routines by predicting them instead of having to entirely rely on direct commands or manually programmed routines. It can also potentially make power usage more efficient as to only restrict device usage to when they are needed without compromising inhabitant comfort.
Smart homes usually integrate sensors and smart devices connected over a network for control, monitoring and recording. These sensors could be device sensors that detect events on devices like switching them on or off. There are also wearable sensors that could be measuring vital signals or user location or activity. Finally, there are environmental sensors that detect things like temperature, light, or presence of a user. Data from these as well as from measuring the power consumption of some devices throughout the day can be logged and then uploaded to a remote server. These data records can be used to train machine learning models to provide better comfort, economising and security features for smart home inhabitants.
Recognising Behavioural Patterns
For a more helpful version of a smart home to be achieved, the home needs to recognize the user’s actions and understand their behavioural patterns or send alerts or suggestions. Predicting user preferences and actions for example could be used to adjust temperature or lighting conditions to suit their activities. It can also be used to save energy like for example turning off unused lights when the user is away or asleep or only turn on water heating when the user is expected to take a shower
An individual user action could consist of multiple registered events. Recognising these complex actions is considered a classification problem where based on the spatio-temporal sequence of events detected via sensors or controls, the algorithm tries to identify the action the user is performing. Long short-term memory (LSTM) networks are applied to recognize these sequences. LSTM is a recurrent neural network that is capable of recognizing sequences within their context as it can retain memory of a past state for an arbitrary amount of time. At applying this neural network to smart homes, raw data from sensors is first processed to create a matrix of sensor events associated with each activity to create classifications. Event history is then fed to the algorithm to be divided and labeled into classified actions.
To recognise and predict users’ behavioural patterns and be able to predict actions, sequential pattern mining algorithms need to be used to find frequent sequences and infer rules from usage data. Some such existing algorithms are PrefixSpan and BIDE+ and an improved version: Gap-BIDE.
Window Sliding with De-Duplication (WSDD) was another unsupervised ML algorithm implemented to treat this problem which is based on a sliding window that goes over the sequence of recorded events in different sizes. And hashing the pattern’s support number (number of repetitions) with the pattern itself as a key. Keys with support numbers above a certain threshold are accepted as real patterns. This threshold is subject to change depending on the threshold providing the highest predictive accuracy. The advantage of this algorithm over the previously mentioned algorithms is that in testing on Windows 8, it has much shorter runtime and uses less memory than the other 3 algorithms⁵.
Another challenge for finding patterns in actions is the presence of a sequence most of which repeats except for a variation of an action or more within that sequence which could be treated as a wildcard so when looking for a pattern, versions of the candidate sequences with a wildcard of length 1 or more should be tested for the presence of more occurrences. This problem is fixed in the WSDD algorithm as it inserts a wildcard action or more in the found patterns and the patterns are scanned again for a matching pattern with the wildcard added.
After forming patterns with sufficient support, these patterns can be used to send suggestions to users with actions that they are expected to perform given their current behaviour. Feedback on these suggestions can be taken from the users to increase support of that pattern or to reduce it. In the tests done in [5], 10 consecutive negative responses resulted in eliminating the pattern.
User Recognition and Security
With integrating machine learning into smart homes, improved security is also made possible. Basic technologies for access control include biometric measures such as using facial recognition. Many facial recognition neural networks exist in literature but integrating this technology in a home environment introduces many challenges: home environment indoor or outdoor (for front door access, for example) has highly variable lighting conditions and backgrounds therefore the technique has to be robust to maintain accuracy within these conditions. Other challenges include cost and time and memory efficiency.
One proposed solution⁷ was using cascaded detectors that process on different levels of refined detection starting with the coarsest — simply recognizing the presence of a face- and ending with fitting the shape of features and finding a candidate face within the stored database of faces.
Beyond security, this can also be used to recognize different inhabitants of the same home to profile their actions individually and lean different behavioural patterns for different users.
Another method to recognize threats is to use learned user behavioural patterns to detect anomalous behaviour which could be a sign of user health or safety risk. A method for doing this is using a decision tree that assesses action with comparison to learned user behaviours and routines. The further down the tree an action is, the more likely it is to be an anomaly. This can be used to detect intruders when other methods like facial recognition cannot be used due to obstruction or poor lighting conditions.
References
3. https://machinelearningmastery.com/gentle-introduction-long-short-term-memory-networks-experts/
4. Dahmen J, Thomas BL, Cook DJ, Wang X. Activity Learning as a Foundation for Security Monitoring in Smart Homes. Sensors (Basel). 2017 Mar;17(4) . doi:10.3390/s17040737. PMID: 28362342; PMCID: PMC5421697.
5. D. Schweizer, M. Zehnder, H. Wache, H. Witschel, D. Zanatta and M. Rodriguez, “Using Consumer Behavior Data to Reduce Energy Consumption in Smart Homes: Applying Machine Learning to Save Energy without Lowering Comfort of Inhabitants,” 2015 IEEE 14th International Conference on Machine Learning and Applications (ICMLA), Miami, FL, 2015, pp. 1123–1129, doi: 10.1109/ICMLA.2015.62.
6. Liciottia, Daniele, et al. A Sequential Deep Learning Application for Recognising Human Activities in Smart Homes. www.researchgate.net/profile/Luca_Romeo/publication/332634798_A_Sequential_Deep_Learning_Application_for_Recognising_Human_Activities_in_Smart_Homes/links/5e15dcc692851c8364bae115/A-Sequential-Deep-Learning-Application-for-Recognising-Human-Activities-in-Smart-Homes.pdf.
7. Zuo, Fei & With, Peter. (2005). Real-time face recognition for smart home applications. 35–36. 10.1109/ICCE.2005.1429704.
