Converting Handwritten Math Symbols into Text Using Random Forest
1. The Inspiration: Is it fair to say mathematicians are averse to technology?
2. The Process: How did I tackle this project?
3. The Future: Where do we go from here?
Prologue: How did I get to this point?
My lifelong love for math inevitably led me to an undergraduate study in mathematics. Soon after taking my first college statistics course, I realized I also had a knack for understanding and interpreting data, as well as coding in the programming language R. After graduating with a Mathematics B.Sc., I became a high school teacher. Even though I can truly say I enjoyed what I did, I still felt the need to search for a more technically challenging career path.
Thus, my search led me to various tech conferences and eventually, a data science conference. I knew right away a career in data science would use my math, statistics, coding, and communication skills as well as natural curiosity in a way that I wouldn’t find anywhere else. This is how I found Metis.
Now here I am, at the end of a grueling 12-week program, mentally preparing myself for our final project presentations. The anticipation I feel thinking about the recruiters that will be in attendance is at war with the excitement and pride I have for completing my passion project. This will be a far less stressful account of the steps I took to complete it.
Note: My code will be in Python since that is the preferred language here at Metis.
1. The Inspiration
As a mathematician, I know how tedious it is to type equations using programs such as LaTeX or Word since my undergrad capstone paper had to be written using LaTeX. I was equal parts frustrated by the sheer effort it took to write statistical equations and amazed at how beautiful they turned out. All the same, I still found myself wishing I could just write the equations by hand!
Did you feel a sense of dread while watching me type this? Yeah, imagine being the one TYPING IT!
All that code, just for the arithmetic mean. Worth it?
I know there are many out there who believe we, as mathematicians, are averse to technology and stuck in our ways. But I would argue, we simply haven’t been given a good enough reason to put down our pencils, or EXPO markers, in exchange for technology. With the tools that we have at our fingertips, such as tablets and smartpens, I believe there is so much more we can do to advance into the future of writing for all STEM fields. As a new data scientist, I decided to take a step in that direction.
2. The Process
I found a remarkably clean Kaggle dataset that provided thousands of labeled images of 43 different handwritten math symbols. After much extracting, balancing, and restructuring, I was then able to think about my next steps. Because the math symbol images were 2-dimensional, 50x50 matrices made up of 1’s and 0’s, instead of the common 3-dimensional RGB image, I was able to .flatten( ) them into numpy arrays. This allowed me to use the 2,500 binary digits as features that would be used to train my model of choice to predict the correct symbol. After splitting my data into train, validation, and test sets, I was ready to begin modeling.
I decided to use a random forest classifier and after some tuning, my parameters were set to ‘n_estimators=500’ and ‘max_depth=9.’ This gave me an accuracy score of 0.75.
Note: I also checked for overfitting which gave me a score of 0.80. Not 1, which is good, but also not 0.75. Overall, I felt this was acceptable.
The numbers weren’t bad, but the true test lied in the application of my model to accurately predict an equation. But first, symbols.
Starting with symbols: I used my iPhone Notes app to draw an eight, took a screenshot, uploaded it to my Jupyter Notebook, reshaped the image to fit into the model, and it worked! Here’s a quick view:
So far, my random forest model had only been trained to take in symbols, not entire equations. To see this through, I created a function to implement a sliding window that would parse an equation into its separate symbols before returning the entire predicted equation. Following the same steps I took before, plus the sliding window, I got these results:
Note: A sliding window wouldn’t be useful if we wanted to transform the arithmetic mean equation (seen in The Inspiration section above). This is a base model.
3. The Future
A more robust model would implement a convolutional neural network (CNN) for image classification. This, along with a wider range of writing styles & math symbols, would allow for complex mathematical equations to be accurately transformed into text.
I would love to see an equation writing app that could convert text equations into LaTeX code to make the writing process less painful.
I’d also consider its use in the health industry. How often is it that patients’ charts need to be entered and updated into a database, but reading the doctor’s handwriting makes that difficult? A model that could be constantly trained and almost personalized could be of great use.
Thank you for taking the time to read about my project. All my code can be found here.
Feel free to contact me if you’d like to discuss this project, my experience at Metis, all things math/data science, or have a job opportunity!