A visual analysis of UK number 1s: delving into development
This part of the series takes a look at the development process of my final year project A visual analysis of UK number 1s. It covers every step, including hosting, setup, Spotify integration, the Scrollama library and fighting with D3.
Hosting
The website is hosted using GitHub pages, as this is an easy, affordable and time sensitive method of hosting a project online. Changes are updated quickly, without needing to FTP files.
Project Set Up
The project was initialised with NPM, which creates a file called package.json. This acts as a manifest for the project and includes each of the packages and applications depended on.
One of the packages used is Gulp, a task runner to ease the development process , I created several Gulp tasks:
- HTML: Copies the source HTML files over to the website directory
- SASS: Compiles the .scss files from the source directory into CSS, and combines them into one
main.cssfile which is saved into the website directory - JS: Finds every JavaScript file from the source directory, uses WebPack to bundle them into one file (
bundle.js) and transpile the code from ES6 to be more browser friendly - Assets: Copies the files from the source assets directory into an assets folder in the website directory
- Clean-up: Deletes any extra files that are copied across to the website directory but not needed
- Default: Runs the above functions when
gulpis run in the Terminal. - Watch: Watches the files in the source directory, whenever they are updated the
defaulttask is run - Minify-JS: minifies the
bundle.jsfile in the website directory - Minify-CSS: minifies the
main.cssfile in the website directory - Minify: Runs the above minify functions when
gulp minifycommand is used in the Terminal
The website directory mentioned above is where the project is served from. This is where the files are compiled and minified for optimum performance. This is separate to the source directory, where code is kept for development -there are multiple JavaScript files here which use the ES6 syntax, unminified for ease of reading. The source CSS files are written in SASS, which also allows code to be written in separate files and combined at compile time.
During development, I used Live-server to serve the project files locally. This refreshes the browser every time the code is updated — making development even easier.
Spotify Integration
To allow the user to log into Spotify and access their data authentication, I needed to use the Spotify API. Spotify supports three different approaches to authentication, and it took me a little while to get my head around them:
Authorization Flow
The user is given a code, which is then traded for access and refresh tokens. This authorisation technique was used in the data gathering stage, however, it does not support AJAX requests and should be done server-side, therefore isn’t appropriate for use on this project.
Client Credentials Flow
This is uses the developer’s client and secret keys in ‘server to server authentication’ and is therefore not appropriate for this project.
Implicit Grant Flow
This is a client-side approach to authentication, which does not use secret keys. The user is redirected to the Spotify Authorisation Endpoint, they log into their account and authorise access for the website within specified scopes (such as user-top-read to get their most listened to tracks). Once this is successfully completed, the user is redirected to a URI specified in the application, with an access token.
The Implicit Grant flow was the most appropriate for this project, as it is fully client-side. The code below demonstrates how this was integrated:
function spotifyAuth() {
var clientID = *clientIDString*;
var path = ‘website/project.html’;
if (window.location.host === ‘rushlet.github.io’) {
var path = ‘ci301_data-vis/website/project.html’
}
var url = `${window.location.protocol}//${window.location.host}/${path}`;
var scopes = ‘user-read-private%20user-top-read%20playlist-modify- public’;
var spotifyRequest = `https://accounts.spotify.com/authorize/?client_id=${clientID}&response_type=token&redirect_uri=${url}&scope=${scopes}`;
window.location.href = spotifyRequest;
}The url variable shown uses ES6 syntax to interpolate variables and strings. It uses window.location.protocol to work over both HTTP and HTTPS and window.location.host to work both locally and on the hosted website. The scopes used grant the website ‘read access’ to the user’s details and most listened to tracks, and ‘modification access’ to their public playlists. These are used to encourage the user to compare their favourite tracks to the number 1s and facilitate following the Spotify playlist of UK number 1s.
The user’s top tracks and audio features for each track are accessed using the Spotify Web API JavaScript library. This information is stored as a global variable in a config object.
If a 401 error occurs when accessing the Spotify Endpoint, the authentication is invalid. This is likely due to the access token expiring and the authentication function is called again.
Below is a sequence diagram depicting the flow for accessing the website by logging in with Spotify:
However, the user does not need grant Spotify access to be able to access the website, it just allows for a greater element of personalisation. A diagram is provided, depicting the flow of the login process:
The top tracks returned from the Spotify API are later used to populate a dropdown in the ‘compare’ section.
Preview a track
Throughout the article, there are opportunities to preview the tracks being discussed, available to both authenticated and non-authenticated users. The preview urls are available from the track endpoint, so I wrote another node script to retrieve these and add them to the main dataset (appendix V.D.2).
Playing a track from a URL uses HTML5 and JavaScript.
<span class=”playable-track” data-id=”1AhDOtG9vPSOmsWgNW0BEY”>Bohemian Rhapsody</span>I used data attributes to add the track id to span elements. To indicate to the user that tracks were previewable, I added icons next to the title. Originally, I attempted to make the sound icons as spans using pseudo elements (.playable-track:after). However, I couldn’t update the background image of a pseudo element on click. Instead, I added them to the background of each span element, which changed on click to indicate the ability to stop the audio.
I added event listeners to each ‘playable-track’ which built the preview url, played the associated audio and updated the icon on click. I made sure that if a track is already playing, the current audio will stop and the new track will play instead. If a track is clicked whilst it’s already playing, it stops.
Follow the playlist
Authenticated users have the option to follow a playlist of UK number 1s on Spotify.
This uses the Spotify Web API Javascript library again, passing the user and playlist IDs to the relevant Spotify Endpoint. This returns a success or failure message which is then stylised using JQuery UI and returned to the user.
JQuery UI was used because it allowed an alert to be styled to match the aesthetic of the website. Furthermore, unlike a JavaScript ‘alert’, it does not pause the execution of the scripts currently being run.
Importing the JQuery UI was harder than anticipated. This turned out to be because the import statement required was slightly different to others used. Rather than importing a specific variable, class or function (e.g. import $ from ‘jquery’;) the entire file needed to be imported (import ‘jquery-ui’;).
It also added a lot of bloat the project. So instead of importing the entire JQuery library, I just imported the dialog JavaScript file (import ‘jquery-ui/ui/widgets/dialog.js’;) and the three CSS files needed to correctly style the modal.
Overwriting the default styling of the modal proved tricky. The library used nested styles which meant the selectors needed to be very specific to successfully override the existing styles.
Building the Visualisations
D3
The graphs in the article are each created using D3.
I used the d3-svg-annotation library to add annotations to both the line and swarm charts. A transform effect is added to the graph to pan to and zoom in on the nodes being discussed, triggered by the user scrolling through the text.
The graphs are written as classes, which allowed me to write certain functions and assocaite these with particular graph types, such as adding or removing lines for the line chart. Common functions, for example, zoom and panning, were later extracted into a separate file (chart-utils.js) to reduce code duplication.
Like The Pudding’s The Differences in how CNN, MSNBC & Fox Cover the News, users are encouraged to interact with the charts and explore the data after the content has been introduced. D3-Zoom is used in the ‘explore’ section of both the swarm and line charts, this allows the user to interact with the graph by scrolling in and out to zoom or clicking and dragging to pan.
Swarm Chart
The artists graph (swarm chart) I created by adapting an example online. I extracted the code out into separate files and rewrote a lot of it. Initially, I looked at the source JavaScript for this graph and commented out aspects to see how each part worked and researched the parts I didn’t understand. Doing this, I learnt some important things — such as how to parse a string as a number (d[‘track_count’] = +d[‘track_count’];) and the effect of different types of scale on the x-axis (such as logarithmic vs linear, pictured respectively below).
One of the major visual changes made to this was to use artist imagery on the nodes in the graph, like in multiple articles from The Pudding. To do this, I needed to add the image URLs to the unique artist data set. This involved writing another node script, which accessed the url for largest image available for that artist, and the dimensions of the image. These attributes were then added onto the unique artists dataset.
To set the background image of a node, I referred to multiple stack overflow posts.
As you can see in the above image, these aren’t perfectly positioned, it was a lot harder than I anticipated. I tried numerous approaches to better fit the images to the nodes (examples pictured below). I even tried to inspect the code of the Pudding Articles which used the same design, but couldn’t find a solution. I could have spent a long time trying to perfect this, but had to prioritise other features.
Line Chart
My initial attempt at building the line graph went slightly awry (as pictured below).
I soon realised this was because I had forgotten to format the data into annual averages. The graph was instead drawing every single value for each audio attribute based on date.
I wrote a function which produced the mean average for each audio feature for each year. Once the results of this were used in the line graph instead, it was a lot less overwhelming (below).
To make the graph a true line chart, fill: none had to be added to the lines which was slightly counter-intuitive.
I then added the key, labels and selected the colours based on a flat colour palette.
Bar Chart
Originally, I had intended to create a multi-series bar chart:
However, this proved tricky in practice. After some trial and error (which was ultimately unsuccessful) I decided to use a standard bar chart instead and allow the user to select an audio feature to visualise using another dropdown. This was quicker, easier, and a better use of time.
Annotations
Toward the start of development, I used colour temporarily for annotations (before I had added artist images), but this was difficult to understand.
I then tried to create my own lines and labels to attach to the graph but this was very tricky and didn’t work well. After thorough investigation, I finally found the d3-svg-annotation library. This looked a lot more professional than my attempts so far. However, I struggled to correctly import it into my project. Eventually, I found an example of it being used with ES6 import statements on the Guardian’s GitHub which worked. It then took a long time to figure out how to correctly position the labels. I made use of the debug feature to help me figure this out, but was mostly a process of trial and error.
Eventually, I succeeded. After this, it was easy to apply the same logic to the other nodes on the graph and add annotations to the line chart too.
Scrollama
Scrollama is a tool produced by staff at The Pudding for scrollytelling. I configured it to trigger the next step when the relevant text reaches ⅓ of the way from the top of the page. A switch case checks the data attribute of the text box in view to trigger the relevant d3 transformation and add and remove annotations. An excerpt of the switch statement can be seen below:
switch(currentStep.dataset.step) {
case “swarm — intro”:
chartFunctions.zoomReset(‘swarm-chart’);
chartFunctions.removeAllAnnotations(‘swarm-chart’);
break;
case “swarm — longest”:
chartFunctions.zoomAndPan(‘swarm-chart’, -1950, 150, 5.5);
chartFunctions.removeAllAnnotations(‘swarm-chart’);
chartFunctions.annotate(‘swarm-chart’, ‘Beatles’, 758, 212, 15, 40);
chartFunctions.annotate(‘swarm-chart’, ‘Elvis’, 850, 210, -15, 42);
break;
case “swarm — successful”:
chartFunctions.zoomAndPan(‘swarm-chart’, 225, 75, 3);
chartFunctions.removeAllAnnotations(‘swarm-chart’);
chartFunctions.annotate(‘swarm-chart’, ‘Cliff Richard’, 500, 218, -15, 33);
chartFunctions.annotate(‘swarm-chart’, ‘Madonna’, 320, 205, 0, 46);
chartFunctions.annotate(‘swarm-chart’, ‘Westlife’, 250, 180, 15, -20);
break;
}I chose to use Scrollama as it seemed right to use the tool developed by the organisation who have provided this project with the most inspiration. Furthermore, the documentation was thorough and easy to understand.
I began by using the example in the documentation and simply changing images on scroll to ensure it was working. I then added a D3 graph and changed the background colour of the container on scroll to ensure it could be combined with D3.
Dropdowns & Personalisation
The final section of the website allows an authenticated user to compare their most listened to Spotify tracks to the number 1 songs. If the user is not authenticated, they can choose two number 1s to compare.
Users select the two songs to compare from a dropdown. Due to the number of number 1 songs available to choose from (over 1300), a standard dropdown wouldn’t be appropriate. A searchable dropdown input, where the user can select or search for an item was a better solution. To create this, I used the Selectize library.
The value of the dropdown option was set to the Spotify ID of the track. This allowed for easy retrieval of the full data from the dataset for each selected entry. This data is then formatted to only include the relevant fields (title, artist, select audio features) before being passed to the bar chart function.
The Selectize library was difficult to import and added a lot of excess files. Rather than using NPM to install the entire directory, I found the two relevant files (the core CSS and JavaScript) and added these to the project locally. To indicate this, I added a libs folder in both the CSS and JS source directories, where I saved the files.
Design Decisions
Design was an iterative process, adapted during development due to the timescale of the project and in response to issues which arose. For example, originally, I had planned to create a gantt chart to illustrate which songs had lasted longest at number 1 over time, showing the difference between consecutive time at 1 and songs re-entering the top spot. I decided not to do this — partly because my initial research had indicated that 3 charts were optimal in an article but also because the charts I had created met my objective of starting to learn D3. Omitting this chart would also allow greater time for testing. However, I still wanted to include the information for this section, so I wrote out the text I had originally planned to include with the click to play songs to make it more interactive and engaging.
And there it was, my working project! It took a good few months and a lot of effort, but I was really excited to see what I had managed to achieve — and learned so much in the process.
Back to part 3- Digging into design
Continue to part 5- The trials and tribulations of testing
