Building a playlist recommender with Vue.js & face-api.js

6 min readJun 26, 2020

At the beginning of lockdown, I took stock of my technical skills and realized they were pretty lacking — sure, I’d previously worked with some web technologies for a work project, and I had been passively following an online course on Swift development, but I didn’t have anything concrete to show for those involvements. This is why I decided to challenge myself to build a full-stack application during the month of May, in order to gain more experience with industry grade technologies (with the added benefit of improving my portfolio of work — you can check that out here).

The result of that challenge was Mood. a web app built with Vue.js incorporating expression detection and the Spotify API. This application was the happy result of me needing new music, as well as the fact that I had previously taken a Coursera course on TensorFlow.js which I hadn’t made practical yet. It takes in webcam input of the user’s face, detects their facial expression, and uses said expression in an API call in order to recommend playlists. It was deployed using AWS S3 & CloudFront, and makes use of Serverless to host the Express.js backend. Currently, it is a proof of concept demo limited to the expressions provided by faces-api.js.

In this article, I detail the implementation of the client and server, and talk about a few of the challenges I faced as I built the app. I also link some helpful resources for anyone who’s interested.

Architecture

Below is the architecture of the application, as well as an initial wireframe I created in Sketch. I find that starting with a wireframe allows me to visualize the hierarchy and interaction of the various components of my pages. The architecture allows the same, except for understanding my requests and the data I expect said requests to return.

The rundown is the user makes a face in view of the webcam, which captures the facial expression and runs the face-api.js models in the browser. These models return an expression, which is mapped to a “mood”, which is then used to query the Spotify API. Spotify returns a JSON response containing a few playlists, and those playlists are rendered on the Vue frontend.

The initial (minimal) application wireframe.

Client Implementation

The client was scaffolded with the Vue.js CLI, and styled with BootstrapVue. One of the biggest benefits of Vue is the component-based architecure it promotes, which allowed me to create components for various elements of the frontend, which the Home view would house the:

Webcam components
Header & Footer components
Playlist component

On mount of the Home object, I created an init method to setup my facial detection and expression detection models. Upon click of the capture button, an event is emitted from the webcam element to the parent, which then calls the method getEmotion(). This passes the captured image as an input to the model, which returns undefined if an expression is not detected.

At the end of this flow, an image corresponds to an expression, which is set as the user’s mood. This mood is then passed as a prop to the recommendation component, which watches for a change in the prop, and makes a call to the Spotify API.

The API returns a json of 9 playlists, which are bound to my Playlist component, and are displayed dynamically using a v-for loop.

Displaying json objects using a v-for loop.

Server Implementation

The server was built using Express.js, and makes use of a node wrapper for the Spotify Web API. It houses a single endpoint ‘/recommend’ — which expects the request query to contain a mood string, and uses that mood in the call to the Spotify Web API.

Deployment

In my opinion, this was the most exciting part of the project, as it was the first time I’d ever built/deployed an application of any kind. I found a helpful guide by Okta which walked through deploying an SPA using AWS S3, which you can check out here for the full breakdown.

I won’t go over the process fully since I’ve linked it, but it involved:

Running npm run build to create my dist folder (static files required to serve a website)
Creating an Amazon S3 bucket for static website hosting & setting appropriate bucket policies
Deploying build directory using aws-cli
Creating a CloudFront distribution

For the server, I used Serverless to deploy the Express backend — the term serverless is used to describe “an app running in the cloud that doesn’t require the developer to provision dedicated servers to run the code.”

This process involved:

Installing Serverless using npm
Creating a Serverless configuration in the server app
Deploying said serverless configuration

This results in an endpoint in the form below, which can then be used in your Vue API calls. ({proxy+} is a wildcard that means that any endpoint with the root /dev/ can be served — provided it has been accounted for in your server code. eg. /dev/recommend, /dev/check, etc.

endpoints:
  ANY - https://YOUR_END_POINT.amazonaws.com/dev/{proxy+}

I made use of a dotenv file to keep track of my endpoints, and made sure that my production code was calling this endpoint.

Demo

Here’s a gif showing how Mood works. I’ll be working on adding progress indicators and handling errors to improve the user experience.

Challenges I Faced

Faces-api Integration

For a while I struggled with the integration of the expression detector, but then I realised from the documentation that I had been running a version of tfjs-core that it wasn’t built for. Additionally, I wasn’t properly handling my errors in cases where no face was detected, so although the model was detecting, it returned undefined (which messed up the rest of the process).

2. CORS Errors

CORS (Cross-Origin Resource Sharing) is a mechanism that allows restricted resources on a web page to be requested from another domain outside the domain from which the first resource was served. — Wikipedia

This topic could have an article to itself — but basically, when building an application, accessing resources outside the scope of said application is not enabled by default. CORS errors usually occur when said mechanism is not enabled, and if you’re using Node, making use of the cors package can save you some grief.

This gets a bit more complex when it comes to Serverless & AWS, but there are a variety of resources documenting the process of ensuring CORS is allowed for a deployed application.

Conclusion

The process of building this application was a really good (occasionally frustrating) learning experience. From my learning the proper way to pass props & communicate between parent and child in Vue.js, to becoming more comfortable checking the documentation & known issues of a library— I know I made the right decision on how to spend my free time in May (after binging lots of Netflix when lockdown began in March haha).

This process also helped me demystify the process of developing an application and increased my exposure to industry standard tools like AWS — something I had constantly read about but never actually tried.

Yes, it was definitely challenging — but the fact that I managed to finally see a project through also feels completely rewarding. My major takeaways from this is the fact that a task that seems overwhelming can always be broken down, and consistently working at it (while avoiding the tendency to overthink it) works wonders for both progress & self-confidence.

Here’s to the next challenge — I’ve always been intrigued by GraphQL!