Hackathon Driven Development — taking Dirtlapse to Production
Hackathons at Propeller
Propeller loves hackathons, it’s a tradition that goes back from when we started and we try to run hackathons as often as we can. Our Australia hackathons, based out of the Surry Hills office, are 24-hour events that span across two days where teams of up to 5 are formed and go head to head to build out something new, exciting or fun. Prizes are awarded (from a surprise panel of guest judges from within the Sydney startup community) depending on viability, impact and peoples choice but overall the purpose is to work with those that you don’t usually and to learn something new.
Sometimes the ideas even make it out to production! What follows is a story of success and the delivery of a new feature ‘Dirtlapse’ all the way through to production and all that is involved. If you want to learn more about our hackathons we prepared a montage video showcasing the latest `billion dollar` ideas from the last hackathon.
Let’s start: what is an orthophoto?
An orthographic photograph (or orthophoto) is a perspective corrected top-down photograph of a large area, similar to the Google maps satellite view, but with greater detail. Each time a Propeller customer flies a drone and processes a new survey on the Propeller platform, we generate an orthophoto of what their site looked like from the sky that day, for example
Many construction projects take months or often years to complete. Over time, dozens of surveys are completed with drones flown over the same site. For some years now, Propeller has had the ability to ‘scroll through time’ on the platform and see how a site has changed and developed, allowing a fascinating comparison and progression view. We wanted to make it easier for our customers to visualize & share the progress on their sites.
The Hackathon: what inspired Dirtlapse?
As a hackathon project, we had the idea to take this ‘scroll through time’ feature and extend it into a timelapse video which could be generated and shared by our customers to show the progress of their projects, such as residential developments.
Our idea was to allow the user to select the time range they wanted to present, add their company logo and generate a video file showing a timelapse of their site’s development as seen from the air.
The plan — using what we already had available
Given we only had 24 hours to complete this project, our goal was to use as many off-the-shelf tools as possible. For visualizing map tiles, we used web mercator tiling scheme, a convention standardized by Google Maps for providing images of the earth’s surface as tiles on a flat plane.
The first issue we ran into was loading geospatially aligned data with a consistent, standardized format that third-party tools (such as Leaflet) already supported. Luckily, Propeller already had an internal service which loaded in customers’ survey data in this standard format. We began our project by loading all the available surveys for a selected site, and integrated with this internal service to create an image url for each survey.
Misusing Leaflet
The next problem to solve was generating a single image from the many map tiles that make up the full view of a customer’s site.
Leaflet is an open source web map renderer with the ability to download and merge many tiles into a single higher resolution image for a given tileset, position and zoom level. However, the idea of this project was an entirely back-end-based video generation. Wanting to avoid spinning up a headless browser just to render our images, we instead took the Leaflet library and modified it to work in a Node.js environment… this was a horrible mistake and it took hours before we had anything stable.
If we were to do this again, spinning up a headless browser for map generation would have been faster & less error prone.
Misusing Ffmpeg
Once we had a collection of several high resolution images, these needed to be converted into a video timelapse with the company’s logo and dates overlaid. To achieve this we used the very popular open source project Ffmpeg to generate a video file, composed entirely of cross-faded images with some static overlays.
The pointy end: request timeouts
Everything was finally coming together! At T-3 hours until presentation time we began to test some longer videos… and everything fell apart.
Our core infrastructure had a fixed 30 second request time limit, and with larger requests our back-end service wasn’t able to download all the assets required and generate the video file within the requisite 30 seconds.
To quickly work around this problem, we broke up the video generation into 2 endpoints:
- One, to request that a video be created and POSTing all the configuration (including information such as site, time range, area and company logo). This kicked off the generation process in the background, stored the state in Redis, and returned a video ID back to the client.
- The client then polled the server via the second endpoint, checking to see if the video was ready yet. Once the video was ready, the server stored the s3 url of the final asset in Redis, which could then be returned to the client next time they polled.
Using feature flags to ship a hackathon project in production
Typically, hackathon projects end up as an abandoned git repository left to slowly wilt into obsolescence. However, wanting to demonstrate this feature within Propeller’s core product rather than as a standalone project we decided to build this timelapse feature as a module within our existing codebase. We used feature flags so as to only load this module for particular users (the members of our team) and disabled some of the standard Propeller code quality checks for code within this hackathon module, as we would be the only users to see it.
Productionizing code
A few weeks after presenting, the product team began questioning how much work there would be in actually shipping our hackathon project as a real additional feature within the platform. This was really exciting. There were a few considerations, namely the detailed steps within our typical feature development process that had been skipped due to the nature of its development in a 24-hour hackathon:
- More user friendly interface designs
- Security
- Scalability
- Code quality and maintainability
- Quality assurance
A week was allocated to re-writing the code to match the usual Propeller quality expectations, before the feature was gradually rolled out to customers.
It’s been great seeing Propeller timelapses shared on social media by our customers, promoting their projects! We love that what started as a challenge for our team has become a feature our customers love.
