Introducing Spion IO

Intro:

We built Spion IO, an open source focus group management API, designed to capture page visitor’s session and allow the developer to replay these sessions in our sleek and interactive open source interface. One of the features that we placed the most importance on when first white-boarding our API was the idea of making it free, scalable and in the hands of the developers and open source community as much as possible. As such, we have separated our API into three moving parts that are easy to implement, such as simply adding a script tag.

Creating a Free and Open-Source Architecture:

The simplest way to construct the API would have been to send all page visitor data to our own central database, but that would mean that the service couldn’t be free as we would have to charge for server usage and increased bandwidth costs. Our solution, Spion IO, consists of three components:

1. A script tag that can be injected into a developers HTML that adds event listeners for clicks, scrolls and mouse movement and sends event data to the database
2. Server code that will store that data in the database
3. An interface that access the event data in the database and plays back page visitor sessions

We wanted to give developers the ability to set up their own database that could run on their sites existing server. We also wanted to ensure that it would be easy to implement. The injected HTML and the video-like playback interface need to access the same database. To do this, we created a simple configuration shell script that will install MongoDB if they don’t have it already, inject client and server dependencies and configure all three parts so that they can communicate with the same database.

Optimization:

One of the early challenges we faced was the additional expense of adding many different event listeners onto a developers website and how to handle the data sent from frequently triggered events, such as mouse movement and scroll movement which happens on a single pixel change. The problem with data frequently being sent traditionally is that HTTP requests are extremely expensive. If we were to write a script that could make 10 HTTP requests per second, this would drastically slow down the developer’s website and the page visitor’s experience. As a result of this challenge, we chose to design our script to use WebSockets, throttles and local storage to mitigate the burden of our script on a developer’s website.

How WebSockets Optimize our Script:

First, once the developer has injected our script tag into the desired HTML pages, every page visitor to these websites will begin to, on page load, start collecting data containing page visitor mouse positions and scroll positions. Assuming that the developer has already run our shell script to setup Mongo and configure the models for storing our data into the database, our script will also, on a page load, open a WebSocket connection, reducing the large number of HTTP requests to only one request. This will already improve the performance of our script but we can do better. Furthermore, we can implement every computer’s local storage with the setTimeout method to store an array of page visitor events and send them off in bundles every couple of seconds. By combining all three of these strategies to reduce the amount of requests and frequency of requests, we are able to run our script with little to no interference on the developer’s website.

Seamless Video-Stitch Experience and Website Mimicking:

Now that user event data has been gathered, it needs to be redisplayed back on the platform website in such a way that it does not affect the website. So we cannot just inject a developer’s HTML and CSS. The page’s data needs to be injected into an iframe as an iframe does not affect the content around it. This allows us to create a responsive and interactive React interface and inject a developer’s website exactly as it is into our interface. Due to security implications of iframes, it is difficult to inject CSS (i.e cursor animation) from cross domain webpages. So we had to extract the developer’s exact HTML and additionally also requests to any linked scripts/stylesheets to that page. Our injected script parses through the developer’s HTML file for scripts and link files, instantiates child processes to get the exact content from each of these links and, then finally saves all relevant information to one model.

Lastly, when we have to redisplay the developer’s site on our interface, all of this information is injected into the mounted page such that no cross browser request is made. After the process of mimicking the web page exactly as it was to the page visitor, we further have to mimic the cursor inside the browser in such a way that is would represent the user’s exact cursor positions on the client’s browser.

By iterating through the event’s array that we stored earlier and interpreting each event in the object, we animate the rendered cursor to the corresponding position. Yet, the problem lies in how to give the interface of the animation similar appeal to that of a video interface. Key UI features included a slider to move forward and backwards through the data and rendering that single frame, similar to a video. This was solved by getting to the new event object data only when an animation of moving the cursor has finished, as without it the animations are finished afterwards.

Conclusion:

As proof of concept, we would like other developers to try our library and leave feedback on how we can improve the project or even build upon it. Come check us out at our website and GitHub.

Spion IO is Mustafa Khan, Miranda Monroe, and Jerry Jong