Managing Digital Chores: A helper to separate work from life
Since the beginning of the pandemic, we were asked to work from home. We were allowed bring equipment from the office as well as purchase some equipment to best repurpose the home as a working environment. Working from home does have its pros and cons. There are obvious benefits like saving money on travelling, eating out and work attire. But there are also issues of separating work and home life, and the never-ending online meetings. The lines between work and life have been blurred.
Online meetings can be a chore and cause “Zoom fatigue”. With the constant background noise, unstable internet, muting/unmuting microphones and ensuring your room reflects positive perceptions of you, digital meetings have become a chore.
As a Creative Technologist for Hitachi, I look at designing ideas practically by building and testing more tangible experiences. I had this idea of reducing the cognitive load of digital meetings by intelligently understanding context better. As winter arrived, many in our office caught mild colds resulting in visible and audible symptoms (coughing, sneezing, etc). During calls I found myself often having to mute and unmute my microphone manually, then cover my mouth to clear my throat, cough or sneeze. If you are presenting on different screens or multi-tasking, this required additional work to conform to online etiquette. I questioned why this couldn’t be done automatically. My computer is already watching me, this seems like an obvious thing to do.
My Solution
Thinking about how to do this, the first thing that came to mind was to train a model to detect my gesture of covering my mouth. You can do this relatively easily using Google’s teachable machine, but this would require much more testing and training data to work generally.
My simple idea was to apply face detection only. My theory was that if I cover my mouth, my face wouldn’t be detected and I could use this as a trigger to mute. OpenCV has a method to detect objects using Haar feature-based classifers. There are existing models which available for face detection, so this was what I tried first.
From my quick test, face detection works well and in realtime. I did find the detection fails occasionally when turning my head. This is not ideal as I often turn my head during calls to look at my laptop screen.
I also tested whether covering my mouth would cause the detection to fail, however it succeeded. This is because the model is using eye detection to detect faces.
I tried to look for solutions with better face detection when turning my head as well as solutions for detecting hands. I came across Boris Maysel’s post where he developed a solution to detect when touching your face. His solution had mostly what I needed. It uses MTCNN for face detection. This seemed to be much more accurate as it correctly detected tilted faces. I adjusted when the trigger occurs, and added an AppleScript trigger to mute/unmute the system microphone, as well as a simple buffer/delay to unmute if there is a single detection failure per frame.
The result was fast and worked really well on a Macbook Pro. On the Mac, the input volume toggles between 100 and 0 depending on the mute state.
Download the code here: https://github.com/leoahkun/muteme
Discussion
The automatic muting solution works well and is very useful during my daily meetings. This reduces one of my digital chores and got me thinking how else technology could help me in my new WFH life. I believe smarter ways of understanding context during meetings and assisting users intelligently will help reduce mental load with the increasing digital interactions that we have. This idea could be extended outside of work as we see more cameras entering the home such as on air-conditioners, refrigerators, robot vacuums, baby monitors, home assistants, etc. This raises a few questions.
What other digital chores were created from WFH
How can can understanding physical context help with these digital chores?
How could understanding our digital context help with real world chores?
How will technology in the home change to help you separate work and home life?
If you have any opinions, please feel free to leave a comment below.
