Building the Nexus for Flatiron Field Day
A few weeks ago, the Manhattan campus of the Flatiron School held an event called Flatiron Field Day, a fun little affair in which students got a chance to use what they’ve learned to draw images across a digital mosaic. If you’re wondering what that looked like, check the GIF below!
Inspired by Reddit’s /r/place collaborative social experiment launched on April 1st, 2017, we created and launched our own version of /r/place a few months ago during our first iteration of Flatiron Field Day. And while that first iteration was certainly a success, we of the engineering team that built it felt that there was room for improvement — not only were there a few bugs here and there, but there were vulnerabilities that were discovered by several enterprising students who nearly crashed our server. On top of that, our next Field Day was looking far more ambitious, involving not only Immersive Web Development students from our Manhattan campus, but students from our Access Labs DUMBO campus and students from our Immersive Data Science course. In short, the principal problems we would face were problems of security and problems of scale. In order to tackle both of these issues, we decided to create a middleman between the user’s code and our server to better control interactions with the server. Enter the Nexus.
The Old
In order to understand the role of the Nexus, it is important to understand the architecture of the first iteration of Flatiron Field Day.
Basically, the Node server (hosted on Digital Ocean) stored the entire state of the board within a two-dimensional array of hexidecimal values. The user would install two bits of code onto their machine: a WebSocket/Redis manager and an API that wrapped read/write functions.
The first bit of code would, on initialization, establish a connection to the Node server’s Web Socket as well as request the entire board, storing the entire board on the user’s machine inside Redis — any changes to the board broadcasted via the WebSocket would change the data inside Redis.
The second bit of code was an API containing a set of functions written in either Ruby or JS (whichever language the student was more comfortable with) for reading from and writing to the board. The board would be read directly from Redis, while writing involved sending a POST request to the Node server.
In short, each user had a local copy of the board stored on their machine which was always in tune with the actual board due to the Web Socket connection. Rather than having to constantly send GET requests directly to the server, a user’s client would instead read directly from Redis, thus decreasing the traffic to the Node server. Writing to the board inevitably required a POST request to the Node server, as these changes would need to be broadcast to all connected users.
In order to rate limit students, the provided API would queue the students’ requests, using a basic timeout between requests in order to throttle the request rate to just a few per second. Also, in order to track who was sending the requests for data logging purposes, IDs were given to each team. These IDs would be taken upon initialization of the API and sent along as an identifier with each write request.
The fundamental flaw with the client was that the user had easy access to basic configuration data for the client. They had complete control over what IDs they could enter, as well as access to the configuration data used to control the rate limiting functions. At first it wasn’t a problem: students were too focused on figuring out how to use everything and trying to write scripts to read/write cool stuff to the board. Eventually, some adventurous spirits decided to wander into the code and start playing with values. As a result, we soon saw huge swaths of colors appear on the board and were unable to identify who was doing it.
In a way, this was something we were hoping would happen — there was in fact an award for anyone who could “hack” our code and hijack the rate limiter — but we were all in agreement that it had been far to easy to “hack” and that “nice job, but please stop or you will crash our server” would not be an effective way to get students to stop (it wasn’t).
What we realized was that while we had the right idea in exposing read/write functions to our users, we had given them too much ready access to the code. In our next iteration, we’d have to do a better job of obfuscating the code and making as much of the inner workings of the client as invisible and seamless as possible.
The Nexus
What we decided was that each client needed some sort of gateway, a middleman that would take care of all direct communications (both HTTP request and Web Socket subscriptions) with the server. This middleman would take care of rate limiting requests, serving data to the client, and managing the user’s local version of the board. The client would then be reduced to a simple API that wrapped functions that made HTTP requests to the Nexus.
Overall, the idea was to preserve as much of the functionality as before while separating out the juicy internal bits that might be tempting for students to play with. Preserving the functionality was the easy part, involving much copy and pasting from the old clients of Flatiron Field Day v1.0 — really the only new bit was wrapping the reading/writing functionality in Express controller actions.
Since the Nexus would still be run locally on each user’s machine, there were two principal concerns that came to mind: 1. that there would be need to be very simple, extremely minimal setup instructions, and 2. that the code would need to be hidden in some way or other.
Making Nexus Setup Easy
In our eyes, it would be ideal if the user saw as little of the Nexus code as possible. The best circumstances would have the user run a simple terminal command like npm start and the Nexus would get up and running. Set it, forget it.
The problem was that the user still needed to input a team ID that would be used to identify their requests,. The simplest solution would be to just have them change a value in the Nexus’ code, but we would lose the simplicity of running a terminal command without seeing any code. The question became: how do we set a value for the Nexus using only terminal commands. The answer: npm config .
npm config is pretty straightforward. Given a package.json that looks like the following,
// SHOWING ONLY RELEVANT BITS OF package.json
{
"name": "nexus",
"scripts": {
"start": "node app.js $npm_package_config_teamID"
},
"config": {
"teamID": ""
}
}one simply has to run
npm config set nexus:teamID <VALUE>ex. npm config set nexus:teamID 1
Note: In general, setting config values works as follows:
npm config set <package.json "name">:<target key in config> <value>Looking at the npm start script, you’ll notice that after node app.js is a funny looking bit of code $npm_package_config_teamID . As you might have guessed, whenever you run npm start this bit of code will look for the value stored within config with key of “teamID” and add it to the end of the script. If you had set the “teamID” to “1” for instance, running npm start would run node app.js 1 . Technically, we could have asked the students simply to run node app.js and then add their team ID to the end, but the nice bit about using npm config is that the config value persists, so if for whatever reason you need to close the server or come back to it some other time, the config value would still be there and you would simply run npm start to get going. Set it, forget it.
The question now becomes: how does we access this value within the code? Node provides us with a nifty way of accessing script arguments using the process.argv . This is an array that contains each bit of the terminal script stored as elements. For example, if you had run node app.js 1 ,
process.argv
==> ["node", "app.js", "1"]process.argv[2]
==> "1"
Easy!
Hiding the Nexus
While we still wanted a very enterprising student to be able to locate and “hack” the Nexus and change their rate limiter, we wanted to make it more difficult for them to find. Two ideas immediately came up: uglify the code using Uglify.js, and turn the Nexus into an NPM package. Ideally we would have done both, but I quickly found that Uglify.js works with ES5 and below, and while ES6 compatible versions existed, none were reliable. In the future I may convert the code into ES5 manually or use WebPack to transpile my ES6 code into ES5, but given the time constraint, we opted to save this for Flatiron Field Day v3.0. So instead we went with option 2: turning the Nexus into an NPM package.
I won’t write about the specific details about how to create an NPM package, but it’s surprisingly easy, and the NPM documentation is great — all you really need is a package.json , an account on NPM, and you’re good to go.
Basically once it was a package, I simply required it within anapp.js file and used the aforementioned npm start to get it running. It looked something like this:
// app.jsconst Nexus require('nexus-ffd')
const id = parseInt(process.argv[2])Nexus(id)
In this case, Nexus is a closure that accepts the team ID as an argument and kicks off the initialization events (i.e. initializing the Express API, requesting the board from the server, forming a Web Socket with the server), keeping the ID in memory for the entire run-time of the Nexus.
Looking at this, I realized that any student who knows anything about NPM would be able to quickly realize to look within node_modules for a folder called nexus-ffd if they really wanted to mess with the code, so I decided to write a bit of creative code within app.js to scare off the meek.
// app.jslet bodyParser = require('body-parser')
let axiosDriver = require('nexus-ffd')
let asyncTCPController = require('async-limiter')function importMetaData(axios){
let metadata = JSON.stringify(axios)
let unitBase = parseInt(metadata.length.toString(), 4)//Adjusting macros for DNS overload
let adjustedUnitBase = unitBase - 28for(var i=0; i < adjustedUnitBase; i++){
if(axios.DBCONN.driverLoad){
axios.ENCODING(axios.STNDOUT)
}
}
}importMetaData(JSON.stringify({
DBCONN: axiosDriver(parseInt(process.argv[2])),
ENCODING: bodyParser,
STNDOUT: asyncTCPController}))
Basically the code is garbage that doesn’t actually do anything. I had a bit of fun picking out random bits of technical jargon that I though would scare off students. Only two of the lines are necessary — can you spot them?
The Future of the Nexus and FFD
This most recent iteration of FFD (v2.0) was a huge success, and we would very much like to repeat that success, next time across more campuses and with far more students (my poor Flatiron London students weren’t allowed to participate given the time difference). As it stands, the Nexus seems poised to work at scale and worked great even overseas (I used the Nexus to draw a picture of my manager for all of the NYC campus to see from all the way in London).
Improvements would be to write more informative error responses, to uglify the Nexus’ code, and to handle internal failures (e.g. not enough memory when a student queues too many requests) more gracefully.
I believe that when FFD v3.0 rolls out, we’ll have a much more advanced server running (possibly in Elixir?!), capable of handling far more requests and collecting more data. Time will tell!
References
I would definitely suggest reading Reddit’s take on building /r/place:
https://redditblog.com/2017/04/13/how-we-built-rplace/
and check out a time lapse of the entire 72 hour run of it here:
For FFD stuff, check out this time lapse of our board:
and check out my fellow engineer’s blog on how he built the canvas that’s displaying the board:
