DOM Dominance might be too Prominent: Ways to Leave D3, React, Vue, and Others in the Pixi Dust.
I need fast data visualization on my web page. Maybe you do, too. So, it should be no problem, right?
Well, it turns out to be a sort of mining expedition to put the right mix of things together to make it work well. There are all kinds of attempts playing around with the <canvas> element or that use SVG or that do something custom.
But, we don’t need just data visualization. We have to have buttons and text fields and all those things on the web page that allow the user to define things, to provide and get information. We need standard form data management and measurements data management all on the same page.
Should one JavaScript library do the whole thing? Or, does one all-encompassing library start to collect a mountain of features as the two kinds of data management begin to tug at each other in the development space? Soon, the library grows and grows. Finally, we have our features and are left back where we were at the beginning, begging for FAST data visualization.
You would think that such a clear requirement might have been completely resolved in the early days of web browser development. But, the sea of web page code has much flotsam and jetsam from shipwrecked web frameworks.
I have been looking into web page solutions for a while. I started with all the dynamic HTML and JavaScript nonsense some time ago. Was it the early 2000’s? Before that, I had been worrying about the portability of desktop apps. Before that, I was using build systems, called RAD environments. I used Delphi, Microsoft, and RevTalk (now called Live Talk). I even used Java and the Borland IDE for it. I found out about QT back then. I am still a sort of fan of QT, but I am not really into pure QT apps. No, I like real C++ to be part of what I do. But, I am looking at QT every day if only because I use the IDE to write JavaScript for node.js Express apps in order to get stuff up on the web pages as fast as possible (where my time and the algorithm time both figure into what fast is). So, that’s been some progress through package after package, with me grabbing the floating desktop that is itself floating away from something like the Titanic.
In all this hodgepodge, I am bombarded with notes from a world in love with React, D3, Angular, and others. It’s all supposed to make certain single page apps possible. Yet, in the early 2000’s I was already essentially writing in my homemade JavaScript framework. Pages looked great. I was doing single page apps then, just calling the routines that allowed Ajax to be called up. And, I don’t recall using too much code nor code that took ages to come up to speed on. For a while, I used the Open Ajax Toolkit (OAT), which never became widely popular. But, I had floating windows, cool CSS, nicely responsive stuff on the single page.
I also looked heavily into CMS’s and how they are built. There was a PHP craze for a while. I even wrote my own CMS in PHP. But, I liked the rendering to happen more in the browser than in the server. So, my CMS was more of a JavaScript delivery system, while other successful PHP CMS’s drew down all the resources on little servers. That was a while ago and I stopped writing the CMS when the recession hit. But, I was ready to jump to node.js, which I have done gleefully. The issue was then, and often still is, that too much rendering is done on the server. But, even now and with node.js, things like Handlebars are nice but can be misused so as to let the server grind away with every page request. So, as far back as 2006, I was ready for the JavaScript framework for the single page app. And, my stuff and OAT worked well for me. I was just starting to hear about jQuery then. Soon, the sea changed and new packages started appearing in the wild, wild world of the world wide web.
As some of my favorite standbys started to gather cobwebs, I had to start looking around for some other tools. So, I started looking into React, Angular, etc. Also, I did professional projects focussed around D3 for a company or two in San Jose. And, I was truly impressed by the size, weightiness, and careful and thoughtful responsiveness of the packages. The responsiveness was, in fact, the kind that let me know that the applications were doing a lot of thinking before speaking.
But, like others, I kind of like seeing things happen quickly, especially if those things are animations. Also, if the whole purpose of a single page app is to get replacement HTML faster than loading whole pages, it would be nice to see the replacements happen in speeds faster than loading whole pages.
Recently, I started coming up to speed with Vue. I might say that I am rather pleased with the way that Vue presents data updates to the program. It seems to be considerably less verbose than some of the others. And so, I have already recommended one project to make use of it. But, I still have this concern about playing around with DOM, virtual or otherwise. I started thinking that maybe there might be faster packages than ones I had encountered. So, I went looking.
First, I encountered domc. If I had not found the link to some speed tests, I would have not known about it: benchmarks. After a quick examination, I surmised that domc seemed to be doing things right: short library code, simple means to do the variable replacements, a tidy data handling approach. And, I wondered how I might do better.
Perhaps playing around with DOM is more than necessary to get stuff onto displays. If I want to draw a data graph, maybe all I need is to do something old fashioned like people used to do with Bitblts. I had to wonder if there was something that used the canvas no differently than some program form the ’80s and that used hardware, even, so as to make things burn rubber (or make waves).
That’s when I stumbled upon Pixi.js. Pixi.js is a library that knows how to do 2D graphics on the HTML <canvas> elements, but it’s rendering engine knows all about the GPU and what WebGL can do for rendering frames. The results are amazing. However, the minute you want to do some typical web page stuff, like responding to a button click, Pixi.js turns into a development project. There are no libraries for typical web page interactions based on Pixi.js that I know of. But, never mind. Pixi.js resides on a web page that can be totally standard and that can have no idea that a full-up video game is going on in one of the frames.
So, now I am back to the original requirements. I need fast data visualization on my web pages, and I need clear, precise, concise, and simple ways of dealing with data updates for both form data and measurement data.
Well, at least I can take a look at a small set of packages to at least start to think about what tools are best for the job at hand. You never know! I might have found an island of certainty.
So, I want to consider the D3 way of doing things. Then, it might be nice to look at how domc works (do it now before it gets bigger!). And, then perhaps it would be good to see what can be done with Pixi.js.
Code
I set up a little GitHub repository with one example apropos to each of the following sections. You can run the repository with the http-server accessible from npm. That is: npm install http-server
Here is the repository: DOMinance
The D3 Way
D3 was getting a lot of love just a few years ago. Perhaps now it’s in common use. People really like D3 because it has this cool, organized way of moving DOM elements through a life cycle based on the data that is being displayed.
Interesting! D3 manipulates DOM in a well thought out way.
D3 is not using virtual DOM. It is keeping track of the real DOM. Let’s review how that works. Then, we can ask questions about it.
In the example programs provided, I chose to alter a gist made by Allan Smith. He gives a good example of using the enter and exit sets that d3 creates when adding or removing dom elements that are mismatched with data. D3 does a good job of making sure that there are only as many DOM elements as data points being used for creating visualizations.
In the code, you will see a button handler that switches between two different sized data sets. When the data sets differ in size, d3 has more work to do to keep things in Sync.
//button to swap over datasetsd3.select("body").append("button")
.text("change data")
.on("click",function(){
//select new data
dataIndex = (dataIndex==1) ? 2 : 1;
//rejoin data - the SVG group is being revitalized
var circle = svgDoc.select("g").selectAll("circle")
.data(eval("dataArray"+dataIndex)); // This is the majic formula for deleting dom elements that
// no longer live with this data set
circle.exit().remove();//remove unneeded circles // Here is the majic formula for getting a new supply of
// circles to be used to visualize new amounts of data.
circle.enter().append("circle")
.attr("r",0); //create any new circles needed with radius //update all circles to new positions
circle.transition()
.duration(500) // animation time
.attr("cx",(d,i) => {
var spacing =
lineLength/(eval("dataArray"+dataIndex).length);
return xBuffer+(i*spacing)
}).attr("cy",dataIndex == 2 ?
(d,i) => { // helps show off lots of data
return (yBuffer + 5*d*(0.5 -
Math.random())) }
: yBuffer )
.attr("r",function(d,i){return d}); // text and lables
d3.select("text").text("dataset"+dataIndex);});//end click function
So, the above code switches between the data sets. Only, I made one of the data sets have 10K points, and the other contains about seven.
You’d expect to see with your naked eye a sort of pause when going to the bigger set. And, you do.
Picking up the Pace with Virtual DOM: domc
I wanted to look deeper into domc. I tried its benchmark, and I was quite impressed. So, I wanted to find out more.
I picked looking into domc rather than React or Vue because it did the benchmark the fastest and also uses a virtual DOM. I figured that it might be a bottom line for the virtual DOM type of frameworks. And, in this article, I am interested more in virtual DOM in contrast to other methods than the war between different virtual DOM frameworks.
There are two things that a framework needs to do once a web page structure is already set up and ready to go: 1) It needs to update data quickly; 2) It needs to do lazy loading so fast that no one would think to call it lazy.
domc builds up structure very fast from its template JS that is in the page. It has no problem adding thousands of rows of data. But, I wanted to see how the data update would go. So, I morphed the benchmark code example to deal with many more updates than the original.
The speed of updates is great. But, domc is a newly born framework. And, it took some time just toying with it until I gave up trying to change the width of <div> elements. domc had no way of dealing with attributes in the version available at the time the benchmark was made. I tried a new version, but it seemed to fail if it was simply dropped into the same envelope as the tested version. So, perhaps I should know more about it. But, the documentation leaves a little to be desired. All that said, however, it does the benchmarks faster than any other framework. Also, it has a plan for handling data updates.
In spite of problems, the domc has great performance. It uses a virtual DOM for better or for worse, but faster than the other virtual DOM implementations. When it comes to changing all the text in 10K rows, it seems to respond in a speed similar to D3. But, I did not set clocks. This is just a report on my personal visual experience.
The full extent of the application in the test case is about forty lines without the data creation part.
const scope = {
add: () => {
scope.data = scope.data.concat(buildData(1000))
main.update(scope)
}, run: () => {
scope.data = buildData(1000)
main.update(scope)
}, runLots: () => {
scope.data = buildData(10000)
main.update(scope)
}, clearData: () => {
scope.data = []
main.update(scope)
}, update: () => {
const data = scope.data
for ( let i=0; i < data.length; i++ ) {
data[i].indicator =
indicatorLine.substr(0,Math.round(Math.random() * 100))
}
main.update(scope)
}, swapRows: () => {
const data = scope.data
if(data.length > 998) {
var tmp = data[1];
data[1] = data[998];
data[998] = tmp;
}
main.update(scope)
}, del: item => {
const id = item.id
const data = scope.data
const idx = data.findIndex(d => d.id === id);
data.splice(idx, 1)
main.update(scope)
}, select: item => {
scope.selected = parseInt(item.id)
main.update(scope)
},
selected: 0,
data: [],
}const main = document.getElementById('main')
const app = domc(main)
app.rehydrate(scope)
After spending time on just figuring out what works, I didn’t spend any time working through an example on lazy loading. But, I am comforted by the fact that the compiler can be called at any time for any DOM element. So, there seems to be some promise that if domc becomes mature that there won’t be a bottleneck for the single page apps.
Eat my Pixi Dust!
Pixi.js is a JavaScript library from Goodboys. It uses WebGL to render 2D displays really fast.
The coding for Pixi.js is ever so much a classic graphics program. But, the reward is super fast rendering and a lot of graphics effects. The following function definition gets called many times in the example. Each call renders five hundred points and gets called twenty times.
function drawData(data,offset,N, YOffset,color) {
var graphics = new PIXI.Graphics();
graphics.lineStyle(graphLineThick,(color === undefined? 0x667700 : color), 1);
// draw a shape
var zshift = YOffset + zeroHeight
graphics.moveTo(startPoint+2,zshift);
var mover = startPoint;
var j = offset;
for ( var i = 0; i < N; i++, j++ ) {
mover += delta
graphics.lineTo(mover, zshift - data[j]);
}
//graphics.endFill();
graphics.lineStyle(2, 0x550000, 1);
graphics.moveTo(startPoint,zeroHeight);
graphics.lineTo(stopPoint,zeroHeight);
return(graphics);
}The whole program is not as short as the cdom program. But, the d3 program is not much shorter. Perhaps the older style of graphics programming is a little more comprehensible than the d3 style, even though d3 is supposed to be easy to comprehend.
In my experience, the journey to the page was a little surprising in that the graphic seemed to be already drawn as soon as the page came up. Refreshing the page can reveal a short lag. But, if you run the example, you can see the noisy sine wave swishing by the rendering of the 10K points.
Conclusion
The Pixi.js program seems magical. But, it will be hard to find ready-made packages for a lot of standard web stuff, like building rows of records. But, for the effort of making rows in many of the frameworks, there may be times when something like Pixi.js can be just fine.
So, I haven’t found just the package that solves all my problems. But, for JavaScript frameworks, the maturity and wide of use of a framework bears some weight. Still, it helps to find the framework that allows for short programs and good ways of managing data responsiveness. So, now I am thinking of picking up Vue, which I did not spend time on in this post, and of continuing to use Pixi.js for the cool and the challenging, especially when it comes to visualization.
