Attending BuiltWorlds Hackathon
In October 2019, I traveled to downtown Chicago and attended BuiltWorlds Hackathon 2019 with friends at DeepWalk Research, a Machine-Learning-focused startup which I was a part of. We had a fun weekend in the city — we stayed in a nice Airbnb, had some great authentic Japanese BBQ, and lucky enough, we won the hackathon!
Emily and I wrote this article together, we thought that it’d be nice to share what we made and why we made them. We are confident that our solution is a good first step to solving the problems presented, which I will describe shortly.
The BuiltWorlds hackathon is a competition that focuses on solving the “built world problems.” That is anything affecting the world we live in — namely, our cities, buildings, and the underlying infrastructure. Our team already has a general understanding of the construction industry and its practices, so we felt quite confident about this hackathon.
We all strolled into Roosevelt University in downtown Chicago for the hackathon and marveled at how this one building, our main habitat for 36 hours, was nearly an entire college campus. Upon filing in the semi-crowded auditorium, we attentively listened to the three challenge prompts posed by the company sponsors of the 2019 BuiltWorlds Hackathon: Clayco, Autodesk, and Caterpillar Inc. Specifically, Caterpillar proposed the problem of not being able to effectively analyze the usage and wear off their machinery across construction work sites.
Drawn to Caterpillar’s issue and knowing we had some background in image analytics, we chose to accept Caterpillar’s challenge and immediately started talking to the Caterpillar team about potential solutions. This is also when we met Emily, the only person on our team that isn’t already a part of DeepWalk Research — Emily didn’t necessarily have the technical chops but she was very knowledgeable about the enterprise applications and presentation design due to her extensive venture capital background and prior hackathon experience. This complemented Brandon’s skills so in the end, we became a mighty team of 6.
With such a big problem at stake and knowing we had to use machine learning (“ML”) and image analytics, we needed to break the tasks down, based on what data we had. In this case, we extensively looked at existing surveillance video feeds of Caterpillar excavators at construction worksites. Brandon pulled out his erasable marker and wrote on a clear board the main types of the excavator’s actions which we can potentially analyze: moving, raking, and scooping. We decided to tackle one action first, namely scooping, as it was the most feasible for the limited amount of time we had for training ML models.
We collected a considerable amount of Caterpillar worksite videos from multiple sources. Now we had to annotate our image frames, choose our machine learning algorithm, train it on our image dataset, and test our trained ML models. After annotating hundreds of images across multiple videos of construction worksites, Anshul started training our model via YOLOv2. After a couple of hours of training, we started encountering multiple memory and storage problems with the EC2 instances we had on AWS. The problems eventually culminated and unfortunately broke the remote server…
While the team wrangled on the technical implementations, Brandon and Emily started brainstorming business applications for our solution. It was imperative to not only have a working prototype but to underscore its tangible business value to Caterpillar’s customers.
Essentially, the business case boils down to this question: How could we translate basic information like the number of excavator scoops per set period of time into productivity metrics like overall worksite efficiency? A logical metric was the project progress in terms of total cubic yards moved per day. We knew this metric was key for managers adhering to project timelines, as missing deadlines resulted in millions of dollars in construction claims per year.
Another metric that was of high importance to Caterpillar was machine health, as machine breakdowns can cost up to $180,000 per incident. The number of scoops was a factor in a predicted time modal for maintenance and repair. Lastly, we could measure operator productivity based on the volume of dirt moved.
A quick LinkedIn message to Dan Reaume, Chief Analytics Director of Caterpillar, confirmed his interest in being able to collect the raw data on scooping and its associated productivity metrics. Now that we had these 3 productivity metrics, the most intuitive way to visualize them was an online dashboard. Our dashboard would have 4 panels: 3 of the productivity metrics and 1 with the video of our working machine learning algorithm.
As the other team members continually worked on fixing the back-end algorithms, I decided to take on the front-end task. In order to present the complex data that we collected and analyzed, I created an intuitive online dashboard for users to view the data in real-time. I made sure that the online interface is intuitive, user-friendly, and mobile-friendly so that even workers on the construction site can access it easily and see the information quickly. I drew out some wireframes for the dashboard and started bringing it to life.
We got back to our Airbnb not long after. In the wee hours of the night, the team eventually decided to leave behind out trained YOLOv2 models, and switch to Mask R-CNN as time turned into a big concern. Fortunately, it worked like a charm. Although we still hadn’t figure out just what the issues were with YOLOv2, at least we had some working models.
By the end of Saturday, we had completely switched from YOLOv2 to Mask R-CNN for training our ML model. As things started to work again, we celebrated a little with the bottle of (cheap?) wine we were given at our Airbnb… We played some darts, watched some Netflix shows, and had a really good time together. We also wrestled with team names and settled with DeepScoop, representing our affinity and perhaps slight geekiness for Deep Learning. There was no going back now.
On Sunday morning, a mere two hours before noon, we finally managed to have a working video of our model successfully detecting the number of scoops of the Caterpillar excavator. Phillip incorporated the video and training data into our completed DeepScoop dashboard. Meanwhile, Emily had hammered out a sleek pitch deck. With an hour before the 12 pm deadline and knowing everyone had to present, Emily would begin by describing the key metric of productivity for construction companies: project timelines, operator efficiency, and machine health. All these factors were necessary for maximizing productivity and reducing unplanned downtime.
In just 36 hours, we built a solution called DeepScoop, and it uses Machine Learning technology to transform surveillance video feeds into informative insights and empowers machinery manufacturers like Caterpillar with powerful analytics data.
At its core, our MVP solution counts the number of scoops of excavators from surveillance video feeds and analyze them based on time-frequency and other metrics. These seemingly normal videos were transformed into quantitative metrics via our dashboard, allowing worksite workers to view complex analytics data that doesn’t usually get seen with human eyes.
Our final presentation at the hackathon was smooth sailing and we saw that the judges were intrigued. When asked about including other factors in productivity, Brandon answered that DeepScoop could incorporate weather data, the number of on-site construction workers, the amount of building materials, and so much more. Counting scoops was just an early start to solving this big, complex problem — it’s only a proof-of-concept, a barely feasible minimum viable product (MVP) we came up in the very limited amount of time we had.
After our presentation, we all felt very uplifted. We had all put our best, valiant efforts and were happy no matter the outcome. To our greatest delight, we actually won the hackathon! Right before they announced the winner, Dan Reaume said that the winning team “did not bite off more than they can chew by effectively presenting a small piece of the puzzle to a larger issue” — at that moment, I knew they’re talking about us! We were all indeed very happy about this positive outcome.
The great part about the hackathon is that we conversed frequently with the Caterpillar team, learning how they leverage the powerful services AWS offer for their digital services and platforms. We were fascinated by their innovative strategies with analytics platforms, autonomous vehicles, and drones to revolutionize the $10 trillion construction industry.
We celebrated with a meal at Gyu-Kaku, an authentic Japanese BBQ restaurant. We were all very thankful for the opportunity to attend BuiltWorlds Hackathon. I met so many people from different industry areas and learned a great deal this weekend, so I couldn’t be happier. This was the third hackathon that I’ve attended in my college career, and I am confident to say that it’s perhaps the best one yet.
This article was written by Philip Kuo and Emily Kuo in collaboration with Anshul Shah, Vishal Narasimhan, Quang Tran, and Brandon Yates, who were all a part of the team.
