Mapping NYC Restaurant Inspections
From Internal Hack Week Project to Production-Ready App
Today we released Food for Thought, a new app that allows users to explore New York City restaurants through the lens of health grades, restaurant violations, and consumer ratings for the first time.
We prototyped Food for Thought in a five-day sprint, by a self-organized team, for Enigma’s Hack Week. The challenge involved working under extreme deadlines, and collaborating on a team that didn’t include frontend engineers or designers, making the design and execution of this consumer app all the more complex.
New technical territories
Hack Week is exciting because it’s about delving into passion projects and learning new skills. For Food for Thought, Madeline on the marketing team led the app’s initial design, while two backend engineers flexed their frontend muscles by learning and developing in React. Backend engineer Yang hadn’t worked on frontend development for roughly ten years, “Building the app in React was a great way to dive into modern frontend development…Despite having a bit of a learning curve, React’s opinionated approach to state management meant that once we got it to work, it would work really well when we plugged in all our data.”
The team also dove into the previously-unfamiliar world of data preparation. While Enigma’s Data Engineering Squad typically (and expertly) handles data prep, technical writer Peter and software engineer Annabel learned Pandas during Hack Week to get the job done.
Their work included standardizing restaurant address formats across New York City and condensing 105 cuisine types into 58 user-friendly categories. “There was a lot that went on in terms of transformations and augmentation to make the Restaurant Inspections data useful for the application,” explained Peter.
Entity Resolution on ~ 24,000 restaurants
One of the toughest challenges was matching restaurant entities across New York City Restaurant Inspections data and Yelp data, as the datasets do not share a unique identifier. Entities needed to be resolved so that the correct Yelp data was pulled in whenever a user selected a restaurant in the app. This allows a user to view inspections data as well as the attributes from Yelp such as consumer ratings and review count for a given restaurant.
Restaurant addresses worked as a linking attribute across the Inspections dataset and the Yelp dataset, but there were many instances in which addresses and names were not identical for a given restaurant. In those instances, the team pinged the Yelp API with a restaurant name and street address to receive a list of restaurants within 50 meters of that geolocation associated with the address. From there, the team reviewed that list to see if a restaurant had the same or a similar name, and if so, would assume a match.
Deeper familiarity with Enigma products
The team also did their fair share of dogfooding. Food for Thought is the first live app to run on the Public API, as the team used the Enigma Public backend to power the app’s entire search experience. “The Enigma Public API is pretty powerful, in terms of what you can search, which was eye-opening to me the first time we hooked up the app to Public. It was really cool to see results come back,” noted Shuo, a frontend engineer who joined the project after Hack Week.
Additionally, Peter and Annabel leveraged Enigma’s data operations platform to build and run all the pipelines that deliver data to Food For Thought. “In this case, it was the ideal infrastructure,” noted Peter. Through the platform, the team worked on individual pipeline tasks in parallel and then compile those tasks into a single workflow. The pipeline now runs unattended on a high performance computing cluster, which is how the Restaurant Inspections dataset remains current.
In conclusion
The Food for Thought team members were invigorated by their rapid-fire experiences learning and developing in new libraries and technologies. By stretching into new areas of development and data problem solving, the team expressed that they felt new levels of empathy for their colleagues that work on similar challenges every day. Annabel summed up the team’s feelings, “It was great to try all these different tools and understand why people love them so much.”
Explore the Food for Thought app and let us know what you think!
__________________________________________________
If you like working on hard problems with smart people, check out our open roles — we’re hiring!
Originally published at www.enigma.com on July 11, 2018.
