How Did I Build a Google Assistant To Check My “Physical” Mailbox?

I am a developer, and I am a lazy guy, you know, so I am always looking for ways to improve my life by making the computer doing the work for me. This article is the story of how I utilize existing services/APIs mostly provided by Google to make a Google Assistant Agent to check my mailbox every day. I repeat, not the email mailbox, but the physical mailbox.

The Use-case

Every day I get home from work, I park my car in the garage and have to walk back to my mailbox, which is in front of my house (but still), to get the mails. Most of the time, there are more advertisement and marketing mails than the important ones. I don’t know why USPS still hasn’t implemented something like CAN-SPAM Act to prevent people from sending junks physically without my consent. So I am set out to solve this situation as a weekend project.

So here is my use-case. On the way driving home, I would like to know if I should stop and check the mails for today for important mails. If they all are spam mails, I would skip and check again tomorrow.

The Solution

I make use of USPS Informed Delivery service as a starting point. If you don’t know about USPS Informed Delivery service, I would highly recommend registering to it. Basically, every day around 8AM, USPS sends you an email with the pictures of incoming mails for the day.

What I did was using Google App Engine Cron Service to trigger a function in Google Firebase Cloud Functions. This function uses Google Gmail API to search and download the email from USPS, extract the texts using Google Vision API and save them to Google Firebase Realtime Database. Then I create a Google Assistant agent to answer my voice queries. The figure below explains the data-flow of the entire solution.

  1. Around 8AM every day, USPS send me a message with subject “Informed Delivery Daily Digest”
  2. Google App Engine Cron Service runs from 8AM to 2PM which triggers a Firebase Function
  3. The Firebase Function uses Gmail API to search for the email with the same subject, download all of the attachments.
  4. The Firebase Function submits all attachments to Google Cloud Vision API to perform text detection.
  5. The Firebase Function uses the texts to categorize important vs. spam mails. The logic is currently as simple as if the mail’s recipient is myself, then it is important. If a mail is sent to “current resident,” or “occupant” then categorized it as spam. Then the Firebase Function saves the attachments with texts to the Firebase Realtime Database.
  6. I speak to the Google Assistant Agent to ask about the mails. Google Assistant interacts with Dialogflow.
  7. The Dialogflow agent then talks to another Firebase Function to fulfill my requests.

Demo

This is the initial version My Informed Delivery (please ignore the agent logo):

Future Works

  • Recognize the sender and use it in the logic of categorizing mails.
  • Use a Machine Learning model to categorizing mails as important or spam more efficiently and accurately.
  • Improve the Google Assistant dialogs.
  • Clean up and publish the code as a public repo.