Image Recognition, Auto-Tagging and Machine Learning — Plus Crayons
Posted on behalf of Shawna Paradee
Amazon Rekognition API uses image recognition and auto-tagging to identify the objects, people, text, scenes, emotions, and activities in images and media content. In machine learning, Amazon SageMaker service allows developers to create, train and publish machine learning models. These are both areas in which I am interested in but had yet to discover a way to incorporate this amazing technology into a current project.
Periodically, my six-year-old daughter wanders into my home office. On one particular afternoon, she plopped down with her box of crayons while I was developing a web interface. As I continued to work, my daughter decided to draw what I was building. Suddenly, I wondered, wouldn’t it be great if I could just doodle my UI with crayons?
Reconciling the Rekognition API and SageMaker’s machine learning, I was able to identify specific objects in images and map them to Material Design components. I found that I also needed an optical character recognition service to extract the text from my pictures and interpret them correctly to code as well. Amazon’s Textract OCR allowed me to not only retrieve the text and its location, but also had a tagging technology that successfully connected to my machine learning model.
I was absolutely astonished by my results. As long as I followed a few simple rules, and did a generally decent job at drawing the components, my solution allowed the conversion of a UI drawing into a Material Design UI template HTML file and companion CSS. Then, I simply take a picture of my drawing, add it to my Dropbox, and my solution takes it and converts it into code. Below is the UI I developed using this method:
Learning and leveraging emerging technologies is a constant process for developers. Innovations and inspirations come in unexpected ways but can revolutionize our business and daily lives. I never thought crayons would make their way back into my toolbox, but I am excited to see how this may change UI development.
Figure 1 below, is a diagram of the workflow and application logic:
Other than the required licensing for the Amazon services, the code is available publicly, and I have given it an MIT License. The Amazon services used all provide a free trial period, so that you may test this solution out for yourselves, before having to pay a subscription fee.
One surprising outcome of the machine learning algorithm was its ability to interpret images meant to be icons. I included mappings to AwesomeFonts and GIF libraries, and the result of my community workspace was funny and impressive. They are easily found and deleted, should this be an issue in your place of work.
Lately, I’ve had trouble getting my daughter to share her crayons. The solution works best with markers and colored pencils, but I personally favor crayons. I had to develop a particular fuzzy logic to assist the OCR to interpret characters in crayon accurately, but it was worth the time and effort. See my repo on how to set the build for each option: crayon, marker, pencil.
So developers — grab your crayons, markers and colored pencils, and look for my repo on GitHub: Crayon-Material-Design.
