A Different Approach to Developing Recommendation Engines using A.I
As a CEO at Myrl Tech, a huge part of my job has been analyzing the needs of the startups that come to us with digital product development requirements. Since it’s 2018, most of the startups are bitten by the A.I bug and want some aspect of M.L and A.I built right into their product from day 1 itself.
The most common way of implementing A.I into your product, still remains the recommendation engine as it has various applications in a product or a platform as all platforms revolve around users and all users want to be able to see more of what they like, and hence, the recommendation engines.
The garden variety approach of having python do all the heavy lifting does work out when you have a lot of past data stored into your databases (like hadoop), especially if you want to query everything offline, create relationships and then store them back in the database. But what if, you are really a new product and have less data to work with and what if, you want to build something specific to the web and also, what if, you want these recommendation engines to work ‘real-time’ (yes, you read that right, real time) rather than post the fact (after your session expires and whatever you did gets stored in a database, gets analyzed and the platform has some content ready for you when you login the next time, similar to what you had viewed / searched in the last session).
We’re talking about a different approach here. Something that lets you define the relationships between the data right from the beginning itself rather than run post-the-fact operations to connect data. We’re also talking about being able to access these relationships real-time and also adapt these relationships based on new data, giving us a lean mean, connected to our web app / mobile app recommendation engine that works with very less data and based on pre-defined data relationships without running heavy python queries on heavy data sets and all happening within a split second. So, let’s see how this can work.
The new approach to this is using Neo4j, NodeJS and Firebase, all of them in your back-end and using React and Redux in your front-end. Now, we all use Nodejs and React together (that’s bread and butter, right?), but what’s up with Firebase here and what on earth is Neo4j?
Neo4j is a state of the art database technology that goes beyond relational (SQL) and non-relational / scalable databases (no-sql ones such as Mariadb, Mongodb etc.) and defines data in sort of a graphical manner where each data point is called a vertice and are are able to define various complex relations between the data points. How does this fit in or help? Ok, good question, So, we have used SQL in applications that require strict relationships between data like an employee database, patient database etc. but developers soon started developing applications that were beyond simple employee databases and required bigger data dumps that would many times not conform to a particular standard and this is why non-relational databases with concepts such as ‘collections’ started getting used. Developers soon realised that the scaling problem and data conformation problem is now solved, but what if you wanted some relations between the data? this is difficult to do with a no-sql / non relational database as it wasn’t meant to do this and if you start using no-sql database to create sql type relations between your data, you are clearly missing the point and increasing development time of your application.
This is where Neo4j comes in to help. Imagine a database that could hold big dumps of data but also have defined relationships like SQL but only, the relationships could be way more advanced (we’re talking hundreds of complex relations between data points). Take a look at the image below on how we define structures in neo4j -
Neo4j — defining relations and data usage with recommendation engines powered by artificial intelligence
Did you see that? awesome right? you’re getting a neural-network type data representation right at the database layer, saving you loads of time, algorithms, re-arrangement, sorting and storing of data after finding and defining relationships between them. Now, what was all that talk about changing relationships in real-time? This is where firebase comes in, firebase (if you don’t already know) is a real-time database technology created and made popular by google and has various uses in applications that run real time (such as taxi booking where a driver can indicate when he is on duty or off duty and chat apps where users can indicate when they are offline or online). What if you could store the real time searches or videos being watched by a user in firebase (the real-time database) and use these relations themselves to show similar videos (by making firebase interact with neo4j) and firebase can also be used to note down what type of videos are being watched by users after watching a particular video and then push these relations to neo4j (firebase can run a script that can programmatically push these relationships or define more relationships in neo4j).
We will use NodeJS in the back-end to create APIs and express to help route the entire application. NodeJS is also the binding gum that holds this complete architecture together. React is used in the front-end because it doesn’t require a page refresh to work its magic (namely, changing states of the components and displaying new data) thus making our application truly real-time and working asynchronously without the need of a page refresh. Image below for architectural references —
Notice, how we haven’t talked about A.I algorithms or techniques till now. Well since we have already implemented this structure from bottom-up, you wouldn’t require a lot of heavy algorithms to do the searching, linking and sorting operations as you would otherwise have had to. In some cases, this structure would work so well, by default, right out of the box, that you may never even need special A.I algorithms to pull off anything.
Let’s summarise — You can use neo4j, an awesome cutting edge graphical database technology to setup relations between your data and you can keep adding relations to the data as and when you come across new user queries. For a particular user, you can store his information in firebase and let firebase interact with neo4j to determine what to show to him based on his searches in the same session, then you push all this information to react asynchronously which displays the information without reloading thereby giving a complete real-time A.I feeling to the application.
This article was originally written on — http://www.designandtech.co/
We at Myrl Tech, build some of the most complex digital products for our clients. If you have any product development requirements (especially the complex ones) and you don’t know where to start, well we’re the guys !
Follow for more informative articles.