Untappd meets Neo4j
Beer Recommendation Engine with Graph Data
It took a while for me to explore a topic thoroughly enough to confidently write a post about it, but the time has finally come. Like most youngsters in their 20s, I’m a big fan of beer — craft beers in particular. Oddly enough, my friends and I like to track our not so carefully chosen beers in an app called Untappd.
According to its’ founders, Untappd is a networking service and mobile application that allows users to check-in beers as they drink and share this experience with their friends. For any beer-lover unfamiliar, I highly recommend checking out the app here. As much as I love Untappd, I’m convinced there is one thing missing: a beer recommendation function.
Besides drinking craft beers, I work as a data scientist at a major consultancy firm where I’ve spent the last year developing a minor obsession for Neo4j and graph data. Graph data seems to be an underrated, upcoming technology with too little attention, also in my professional circle in my opinion. Therefore, I have been working hard to excite my colleagues about graph data.
Since everyone only has a limited amount of time in their days, I needed something new and exotic to convince others that graph data is worth their attention. In Untappd data is stored about people’s preferences, social networks, and visits to venues — which would make the perfect dataset for a graph data project.
BOOM, the idea is born! let’s make a beer recommendation engine from Untappd data.
THE PLAN
1. Scrape Untappd
2. Create a graph database
3. Graph Features
4. Write recommendation functions
So, off we go…
Scraping
I am by no means good at scraping web pages, and thus it’s also not the focus of the post. As a matter of fact, when I started the project I was using python’s requests.get() and it got so messy that it frustrated me so much to pause the project for a couple of months. Finally, I gave in and learned how to use Scrapy.
The scraping started by collecting a bunch of usernames: starting with the users I’m friends with, the friends of my friends, their friends, etc., and soon I ended up with a list of 10k usernames (shall we test the six degrees of separation theory in the next project?). I’ve used these to scrape their stats and check-ins. From the activity I’ve extracted which beers people checked in, how they liked it, where they drank it and from which brewery the beer is.
As a point of improvement, I’m working on putting the scraper in a script form instead of a command-line/scrapy shell. Currently, I’m only able to scrape the 5 most recent check-ins from users. However, when I’m able to run the scraper automatically on a weekly basis, I’ll be able to collect more data about users' preferences, which will drastically improve the recommendation made in a later stage. So, if you’re interested and have more experience with web scraping, feel free to reach out.
Graph Database
The initial graph database’s schema looks like the image above. All the nodes have unique constraints and we have 4 types of nodes and 5 types of relationships. The venue’s location is still ‘work in progress’ as there were some issues with scraping the location of the venue.
Graph Feature Engineering
Since the ultimate goal is to recommend beers to users, we want to make that recommendation as personal as possible. As shown in this research article, the strength of friendships is of great importance in personalized recommendations. Moreover, we’ll assume effects of the social proximity effects; your friends’ habits will become your habits, or in this case; whatever your friends like, you tend to like too. Taking these two assumptions into account, we’ll create user similarity based on the beer ratings they gave to beers and construct a similarity score out of that.
//First let's create an in-memory graph
CALL gds.graph.create(
'user_checkin_beerGraph',
['User', 'Beer'],
{
CHECKED_IN: {
type: 'CHECKED_IN',
properties: {
rating: {
property: 'rating',
defaultValue: null}}}})//Writing the :SIMILAR relationship to the graph
CALL gds.nodeSimilarity.write('user_checkin_beerGraph', {
writeRelationshipType: 'SIMILAR',
writeProperty: 'score'
})
YIELD nodesCompared, relationshipsWritten//Analysing the :SIMILAR relationship we just wrote to the graph
MATCH()-[s:SIMILAR]-()
CALL{
MATCH()-[r:SIMILAR]-()
WHERE r.score > 0.3
RETURN COUNT(r) as above3
}
WITH min(s.score) as min, max(s.score) as max, avg(s.score) as mean, above3, count(s) as total
RETURN min, max, mean, above3, totalOutput:│"min" │"max"│"mean" │"above3"│"total"│
╞══════════════════╪═════╪═══════════════════╪════════╪═══════╡
│0.1111111111111111│1.0 │0.14599892562855618│3168 │162162 │
As we can see from the manually generated stats, the :SIMILAR relationship doesn’t provide us with insightful similarities between users. After playing around for a bit, I figured that it is caused by the limited amount of check-ins I could collect about the users. In short, because we just have the 5 most recent beers, it looks just at people who have similar beers checked in, and not so much about what they rated it. Thus you’ll find clusters in the data where friends who regularly drink together, get high similarities. E.g. my girlfriend and I are 1.0 similar because the last 5 beers we drank together we both checked in. Therefore, let’s try to scope down the huge amount of different beers to a higher level of granularity: the category of a beer.
//Firstly, we create a Category Node from the Beers. In comparison, there are ~24k beers which belong to ~300 beer styles. Bringing this down will make it easier to find similarities between people.LOAD CSV WITH HEADERS FROM "file:///beer_details.csv" AS row
WITH row WHERE row.beer_name is not nullMERGE (b:Beer {beer_name:row.beer_name})
MERGE (cat:Category {beer_style:row.beer_style})MERGE (b)-[:IN_CAT]->(cat)//Secondly, we’ll add :LIKE and :DISLIKE relationships between User and Category, based on the rating they gave and the average rating the beer gets. As a threshold I’ve given a user that rates a beer higher than 15% (compared to the average) likes that category of beers. Similarly for disliking.MATCH (u:User)-[checkin:CHECKED_IN]->(b:Beer)-[:IN_CAT]-> (cat:Category)
WHERE checkin.rating > (toFloat(b.beer_avg_rating)*1.10) and toFloat(b.beer_avg_rating) <> 0
MERGE (u)-[l:LIKES]-(cat)
ON CREATE SET l.strength = checkin.rating - toFloat(b.beer_avg_rating)
RETURN u.user_displayname,checkin.rating, l.strength, cat.beer_style, b.beer_avg_rating;//To test this assumption, we can verify if people have :DISLIKES and :LIKES relationships to the same category.MATCH (u:User)-[:LIKES]-(cat:Category)
WITH u, cat
MATCH (u)-[:DISLIKES]-(cat)
RETURN count(u) as indecisivePreferencesOutput
╒═══════════════════════╕
│"indecisivePreferences"│
╞═══════════════════════╡
│137 │
└───────────────────────┘
Considering we have a total amount of check-ins of ~50k, 137 users with indecisive preferences can be disregarded.
I’ve reviewed the following options:
- Creating a :SIMILAR relationship with gds.nodeSimilarity based on an in-memory graph containing CHECKED_IN, LIKES and DISLIKES relationships.
- Creating a :SIMILAR relationship with gds.nodeSimilarity based on only LIKES and DISLIKES relationships.
- Creating a :SIMILAR relationship with gds.nodeSimilarity based on a LIKES relationship where the score being positive or negative meant whether someone like it.
The only problem with these options is that somehow the existence of the relationship to a category (regardless of it being LIKES or DISLIKES) weighted more than the score. In the image below you see two users who have a high similarity score, but both have opposite relationships to the categories they have in common.
Hence I’ve opted to go for two similarities :SIMILAR_LIKE and :SIMILAR_DISLIKE.
//Analysing the :SIMILAR_LIKE and:SIMILAR_DISLIKE relationship we just wrote to the graph
MATCH()-[s:SIMILAR_LIKE]-()
CALL{
MATCH()-[r:SIMILAR_LIKE]-()
WHERE r.score > 0.3
RETURN COUNT(r) as above3
}
WITH min(s.score) as min, max(s.score) as max, avg(s.score) as mean, above3, count(s) as total
RETURN min, max, mean, above3, totalOutput SIMILAR_LIKE:
╒══════════════════╤═════╤══════════════════╤════════╤═══════╕
│"min" │"max"│"mean" │"above3"│"total"│
╞══════════════════╪═════╪══════════════════╪════════╪═══════╡
│0.0490786655006517│1.0 │0.6635704344923967│116314 │122076 │
└──────────────────┴─────┴──────────────────┴────────┴───────┘
Output SIMILAR_DISLIKE:
╒══════════════════╤══════╤══════════════════╤════════╤═══════╕
│"min" │"max" │"mean" │"above3"│"total"│
╞══════════════════╪══════╪══════════════════╪════════╪═══════╡
│0.0777058551244277│11.4 │2.3484375959567565│105172 │105550 │
└──────────────────┴──────┴──────────────────┴────────┴───────┘
Analyzing the results, we see that we now have a lot more relationships to build recommendations for. A person can either have a similar dislike or like preference with other users. Since we have now a lot fewer categories where we build recommendations on (24k beers vs 300 beer styles) it’s much less likely that we’ll see similarity clusters form in the graph of people who regularly drink together.
After the feature engineering, the graph schema changed and looks as follows;
Recommendation Functions
Let’s explore the first and most basic option. We’ll recommend a beer, that is highly rated to a user that is similar to him (based on the :SIMILAR_LIKE or SIMILAR_DISLIKE relationships), that the person has not yet checked in himself.
//First find the top 5 users similar to myself, regardless if we're similary base on what we like or dislike.
MATCH (u:User{user_username:'Bart_Simons'})-[r:SIMILAR_LIKE|SIMILAR_DISLIKE]->(other:User)
WITH u as Me, other as SimilarUsers, r.score as Score
LIMIT 5
WITH SimilarUsers, Me, Score//Than we'll look for beers which the SimilarUsers have checked in, but myself haven't. And return them based on what they've rated the beers.
MATCH(SimilarUsers)-[checkin:CHECKED_IN]->(b:Beer)
WHERE NOT EXISTS((Me)-[:CHECKED_IN]->(b))
RETURN checkin.rating as Rating, DISTINCT(b.beer_name) as BeerName, b.beer_style as Style
ORDER BY Rating DESC
LIMIT 10Output:
╒════════╤═════════════════════════════════╤══════════════════════╕
│"Rating"│"BeerName" │"Style" │
╞════════╪═════════════════════════════════╪══════════════════════╡
│4.75 │"Erdinger Weißbier / Hefe-Weizen"│"Hefeweizen" │
├────────┼─────────────────────────────────┼──────────────────────┤
│4.5 │"Weizen" │"Hefeweizen" │
├────────┼─────────────────────────────────┼──────────────────────┤
│4.5 │"Weizen" │"Hefeweizen" │
├────────┼─────────────────────────────────┼──────────────────────┤
│4.25 │"Weizen" │"Hefeweizen" │
├────────┼─────────────────────────────────┼──────────────────────┤
│4.0 │"Cloud Scanner" │"IPA - New England" │
├────────┼─────────────────────────────────┼──────────────────────┤
│4.0 │"Wit" │"Wheat Beer - Witbier"│
├────────┼─────────────────────────────────┼──────────────────────┤
│4.0 │"Adriaan Wit" │"Wheat Beer - Witbier"│
├────────┼─────────────────────────────────┼──────────────────────┤
│4.0 │"Chouffe Blanche" │"Wheat Beer - Witbier"│
├────────┼─────────────────────────────────┼──────────────────────┤
│4.0 │"Korenwolf" │"Wheat Beer - Witbier"│
├────────┼─────────────────────────────────┼──────────────────────┤
│4.0 │"Weizen" │"Hefeweizen" │
└────────┴─────────────────────────────────┴──────────────────────┘
Making a simple python class from this with a neo4j-python driver we can ultimately transform this into an API.
Output:
Recommendation: <Record Beer=’Erdinger Weißbier / Hefe-Weizen’ Avg_Rating=’3.65689' Style=’Hefeweizen’>
Recommendation: <Record Beer=’Weizen’ Avg_Rating=’3.38229' Style=’Hefeweizen’>
Recommendation: <Record Beer=’Weizen’ Avg_Rating=’3.38229' Style=’Hefeweizen’>
Recommendation: <Record Beer=’Weizen’ Avg_Rating=’3.38229' Style=’Hefeweizen’>
Recommendation: <Record Beer=’Chouffe Blanche’ Avg_Rating=’3.60963' Style=’Wheat Beer — Witbier’>Thanks for reading! If you’re interested in the code or would like to have a discussion on the topic, feel free to reach out to me on LinkedIn.
