Pretty Muni: Data Visualization in JavaScript with D3.js
https://www.prettymuni.com
A web app that uses D3.js and vanilla JavaScript to display and update vehicles from the San Francisco Muni system in real-time.
New colors each refresh!
Showing public transit vehicles in real-time
D3.js is a great open-source data visualization library from Mike Bostock, formerly of the New York Times.
I won’t go too deeply into code in this article, you can check that out at the Pretty Muni project GitHub. What I will do here is provide a line of reasoning for many of the choices that I have made, both visually and in code. My goal is to show how these considerations influence and guide each other during the process.
Additionally, I try to remain conscious of the costs of deploying and accessing the application. Serve the most people at the lowest price! This extends to the personal cost of user time, where I try to make continuous incremental improvements.
I had fun building something interesting this week, but I’m sure there’s a lot I can improve on. Please send me your suggestions and pull requests!
Enjoy the read!
Drawing the base maps — geoJSON layering, colors, time to first interaction, map sizes
With SVG, the order of the elements on the page is the order that the elements are added to the element. If I tried to fetch all of the maps in parallel, the draw order was haphazard depending which was retrieved first. It’s pretty straightforward to reorder the actual SVG DOM nodes containing the maps using built-in methods.
The map files are in geoJSON format,but I dipped quickly into the world of topoJSON, which promises smaller file sizes and more features. However, my early experiments actually ended up with larger file sizes in this particular case so I decided not to pursue it further at this time.
In regards to vehicle colors, Muni does provide colors for each route, but in my personal experience the colors aren’t really used for finding a vehicle — it’s much more route-name based. However, I could be wrong about that and it is hard to dell without more data. If this were a functionally-oriented application (i.e. for the actual Muni organization) I might have gone with the API provided colors.
I assigned random colors to each of the layers, which ends up being sometimes ugly but often fascinating. It’s also a great way to get people to revisit and refresh the page. It ends up looking sort of like a Fauvist version of Mini Metro.
At first, I explored libraries for procedurally generating sets of related colors. The best library I found only does up to 16 colors. I enjoyed the random results enough to hold off on locking the colors down until a later version. I might stick with the randomness but tune it a little bit with an algorithm to distribute colors as far as possible from the previous colors in the set across all 80+ Muni routes. There is a section later about choices regarding route paths and colors.
Of course, with such an interesting map, users are going to want to zoom and scale. I decided to use native SVG scaling and transformations to accomplish this, instead of re-projecting each data point at every zoom level. The result is a smoother, high resolution zoom. An upgrade to passive event listeners could help, especially on mobile.
The file size of the maps is a problem. Almost 9mb for the streets.json file alone. To my great surprise, the streets geoJSON file comes over the network at only 1.2mb. Gzip for the win! Still pretty big, but enough savings to make the app usable at 1.9mb total for all maps.
Later, I realized that waiting for the streets base map to load shouldn’t block the user from starting with the other base maps. I made some changes so that the streets file loads separately from the other base maps. Once it has arrived, I insert its DOM node into the SVG at the correct level. The result is a much reduced time to first interaction, a better user-experience and a big win in terms of how it felt to use the app.
Getting the vehicle data — XML, JSON, Promises
http://www.nextbus.com/xmlFeedDocs/NextBusXMLFeed.pdf
One utility function I didn’t hesitate to use was an XML to JSON parser. I haven’t worked with XML enough in my career to know all of its foibles, and didn’t want to run into them one at a time writing my own parser. Therefore, I leaned on the crowd and on one clever implementation in particular. I did have to update it to supress some Chrome warnings, but it was only changing one or two lines. This allowed meto parse and process the Nextbus API responses in familiar JSON format.
It didn’t take long to start to see a tangle of callbacks beginning to develop in this part of the app. So in the name of readability, I dropped in an ES6-Promises polyfill and wrapped the Nexbus API requests. Now I can easily get and use the vehicles and route configurations in my other functions without getting too verbose.
Drawing the vehicles —adding circles, headings, colors,text, teardrops & transitions
Vehicle groups consist of these elements: circle, text, heading drop, and heading dot. Routes contain many vehicle groups. When I update data for a route, the vehicle groups are transitioned to their updated locations.
I first visualized the headings as dots within the outer ring of the vehicle circle. While this was aesthetically pleasing, it didn’t translate quickly or at great distances. It was too subtle.
I didn’t want to switch to using a triangle or another hard-edged primitive to represent the bus. The circles were nice and clean. My answer was to put a teardrop shape underneath the circle — the heading drop. I moved the heading dot out further and gave it the same color as the route, which helps distinguish between overlapped vehicles.
Route Paths — Too much color?
I was conflicted about whether to add route paths — everything looked really busy already with the streets, colors and vehicles. Maybe it would be too much. I had to see for myself, so I coded up a couple of methods do add route paths to the map when a user chooses a route in the selector.
At first glance, my suspicions seemed confirmed. There is a lot going on. But the more I used the app with paths and showed it to friends, the more I realized that they do communicate a lot about what is going on. Taking a little bit of opacity off of the streets map helped but destroyed the frame rate and made everything sluggish. Perhaps there is a compromise — I like the look of the route paths with the streets turned off. A monochrome palette for the entire set of base maps could calm things down too. I tried black, white, and grey streets, which all worked depending on the colors of the neighborhoods. Yet there is a certain delight in some of the weird combinations that the random selection uncovers. And refreshing is easy.
I’m torn. For now, color roulette, until I get more feedback :)
Creating the route selector — first, see what’s out there. But sometimes you’ve got to roll your own
The native HTML5 <select multiple> element is a mess. On the desktop you need to press control or command and click to make your selections. The Rolodex-style spinner given to users on mobile by iOS Safari is better, but not by much. Either way, the styles involved can be difficult to write or completely out of your control.
To sidestep these issues and avoid maintaining separate codepaths, I decided that this might be a good place to use a library. I’m a fan of Material Design and have successfully used the Materialize library on previous projects. So I decided to give it a try here. While adding a dependency on jQuery just for this made me cringe, the stylistic advantage of a <select multiple> that renders as a dropdown with checkboxes was surely worth a few extra kilobytes.
This looked better. Everything was hunky dory until I tried it on mobile. Then I ran into a nasty Materialize bug — multiple selects cannot be closed on mobile after they are opened. This is a non-starter. I toyed with the idea of using the browser-default on mobile and the Materialize version on desktop, but that would mean feature detection and maintaining different code paths. With all of that, for little gain, I decided it was time to scrap Materialize and the <select multiple> element altogether.
What I ended up creating is a much more flexible, faster route selector that requires less scrolling and looks the same everywhere. It also provided an opportunity to use the route colors to make a connection between what the users chooses and what the user sees. Larger screens have room for the full route names. The controls are fast, even on mobile.
Bonus: I was able to remove jQuery and Materialize from my project.
Interfaces — GUI glue, responsive CSS styles, loaders, saving, sharing
I wrote some standard JavaScript code to handle creating the user interface elements and binding input events to them. This app is straightforward enough that using an MVC framework like Angular or React really seemed like overkill. Not to mention the likelihood of integration issues with the chosen framework and D3. The native DOM methods are actually pretty useful, if verbose, and you usually know what you’re going to get. I even made the close button sticky without too much fuss :)
Since the maps loaded are fairly large, here’s inevitably a bit of initial user wait time. I chose to use a simple, CSS-based loader that is hidden once the maps are drawn. I’d like to upgrade this stylistically to be in line with the rest of the app. Speaking of styles, I enjoyed writing the CSS for the app. A couple of simple media queries really helped alleviate the mobile vs. desktop differences I could surface in my somewhat limited testing.
Saving an SVG to an image is a bit of a kludge. I did some research and found a function that mostly works, although in the end it is missing some styles that are applied to the entire page and not at the SVG level. I’m going to work on that feature a little bit more before I deploy it.
For sharing, I defined Facebook and Twitter compatible metatags, as well as some Slack-style pull-through tags for really fancy sharers. I flirted with the idea of buttons, but didn’t want to disrupt the visual flow, and there are already a lot of built in ways to share links.
Get real —Cross-browser compatibility, performance
Cross-browser and cross-device testing is an essential but sometimes painful process. It quickly becomes race to apply a patch here, a shim there. What’s a dev to do?
Try to put your time developing for where users are likely to come from. This requires knowing your starting user base, as well as paying close attention to your analytics. Spend some time looking at different reports of browser statistics and compare them to the reality at your organization.
If you have to guess, here are some general observations: Currently on desktop, Chrome is crushing it, with a large spread separating it from Firefox and Internet Explorer. The top 3 browsers on mobile at time of writing have a closer spread — Chrome, Safari, and the Alibaba’s UC_Browser.
These decisions come down to resources and reach. The more you have, the can do. Apportion wisely. As far as what I was able to test with Pretty Muni:
On desktop Chrome, things work the best.
On desktop Firefox, things work after a quick to update my XML parser to use .value instead of .nodeValue, but my heading drops are ending up rotated in strange places.
On desktop Microsoft Edge, zoom scrolling results in wild scaling of the base maps. Rendering bugs about and performance suffers. I can’t imagine that older Internet Explorer versions must be doing any better :/
Some mobile testing in Chrome led me to the look into the new Passive Event listener API, which I tried to shim into the current version of D3.js, but ran into some errors that forced me to back off for the moment. The feature might be worth spending some time getting working correctly in order to avoid hiccups during the processing of initial geoJSON maps and achieve really great performance.
The largest performance increase would probably be reducing the number of SVG elements that are being drawn and observed. I’ve already filtered out vehicles that haven’t moved and headings that haven’t changed from their respective animations.
Deployment— https, GitHub pages, mixed-content, proxying apis, node.js heroku, DNS w/ cloudflare
My first approach was to serve the app completely from the client side using a service like GitHub pages or Amazon S3. But when I pushed to GitHub pages, I was greeted with a blank screen and a console full of mixed-content errors. How come?
You can’t use http:// assets on a site served over https://
Simple as that. Production ready code should be served securely. Therefore, the requests to the Nextbus service over http:// were conflicting with the browser’s internal safety barometer.
Nextbus isn’t likely to change their API before I need to deploy (although if they do add https:// the whole internet might do a cartwheel). Therefore, I’ve got to figure out a way to secure my request.
Fortunately, writing a proxy in node.js is a matter of just a couple of lines. The server takes requests to Nextbus and passes them along. To be fair, this proxy server is a bottleneck. It ads an additional server step to an app that is otherwise completely client-side. It also makes it look to Nextbus like there are an awful lot of requests coming from the IP address that belongs to the app. So far they don’t seem to be throttlers, but we’ll see. After pointing the Nextbus urls in my app at the proxy, the page proceeded to load as expected.
I tried deploying the app on my own static server on Heroku using Express, but worried about co-locating the proxy and the app — too much traffic and the whole thing would go down. Better to keep them separate. I don’t have an AWS account setup right now, otherwise I might have just put everything in an S3 bucket. Besides, why pay for all that bandwidth
I’m an avid GitHub user and eventually I came back around to GitHub pages. I had to name my distribution folder ‘docs’, but I was happy to deal with a little awkwardness in exchange for free hosting. (I’ll try to fix that at some point with a deploy task, likely written in gulp.). Now the app and the proxy server for requests are separate, and I’m not paying for hosting the client-side assets. Woohoo!
I used Google Domains to purchase the www.prettymuni.com domain name, but decided to try CloudFlare as my DNS provider because they offer more routing options and features. I set the A records at Cloudflare to point at GitHub’s Pages servers. The final step in securing the site was adding page rewrite rules at Cloudflare to turn all http:// requests into https:// requests.
Why does any of this matter? In addition to actually wanting the content to appear everywhere without mixed content warnings, security now weighs into the SEO rankings of many search engines.
Some quick delivery statistics vis-à-vis bandwidth:
- 107kb JavaScript files
- 2kb CSS
- 11kb Fonts
- 1.9mb JSON to fetch all maps
- route fetches are 1.1kb per route from NextBus
- ~83kb to get all routes at once
Conclusion
A lot can be improved in terms of performance and I have plenty of ideas for additional features — generative sound, anyone? Really though, I’d be happy to just figure out those pesky heading drop rotations.
But that’s all for now!
