Rendering array of billion of records At 60 f/s in Angular or Vanilla JS
WHY: We at SS8 have developed a web-based data exploration platform to analyze network traffic to detect breach in the network. The product is designed for enterprise, even a small size company (ex: 100–500 employees) would generate (traffic) approximately ~2 millions/second of HDR (enriched metadata records). We collect data for days to years and analyze (Big Data), As an average we need to render (in UI)~4 millions of summarized data to slice and dice further by security analyst in the client side to narrow down to conclude the breach and its impact.
If we render all records in the browser, it will not only becomes slow (even clicking a button would take 4–5mins to respond) but may also crash (if not chrome). So we explored many ways to deal with this data (open source / paid libraries):
Try to render 671100 records of height = 50px using any of the solutions mentioned below!!
- Render datagrid using canvas (see example) — Can’t style, difficult -impossible to customize and can’t achieve much because canvas is pixel based.
- Following solutions doesn’t scale because of browsers max pixel limit:
- Virtual scroll: Few implementations “Virtual Scroll” and “rxjs Virtual Scroll” and both of them doesn’t support horizontal scroll.
- Infinite Scroll: “This partially work” for vertical scroll (scrollbar jumps on scroll) but this is limited to vertical.
Paid libraries are not even worth to mention here.
What is the browser max pixel limit we are talking? This question is answered here and can calculate dynamically too.
A Brilliant Solution:
In the past we used a jquery based datagrid called SlickGrid by Michael Leibman. It was optimized for large data, when we tested it worked pretty well even for 50K records then we explored the source code to learn more about how he managed to get it worked.
Anatomy of technic:
Fundamental idea in this technique is to paginate as user scroll. How can we paginate on scroll? At first we have to assume very high but less ≤ browser pixel scroll limit h as actual scroll height/width. Then calculate the virtual height th by multiplying the total records items to DOM element height/width itemSize (virtual height/width is = all dom elements would have fit if browser have no limits)
Here the [=====]represent the actual/real scrollable height and [- — -]represents all pages n . Total height of all pages n is th = ph * n . Overlapping between pages is cj (jumping coefficient) and is the distance the scrollbar will jump when we adjust the scroll position during page switch. To keep things smooth (user will experience smooth scroll), we need to minimize both nand cj. Setting phat 1/100 of his a good start.
We must ensure the parent container is position: relative; and every child (for each record) will haveposition: absolute; and item height must be fixed (variable height can’t be supported — it’s drawback or we haven’t explored it because we didn’t have requirement).
After calculating all required parameters we need to insert a dummy div element and position: absolute; top:0px; left:0px; and height/width: <<h>>px; where h is actual scroll height/width that assumed in the beginning. On use scroll we need to calculate page and the offset required, finally calculate the start and end index of the items/records we should render.
Angular 4.x version demo
CAUTION: Don’t increase the size in the plnkr it will crash, I strongly recommend to create angular project to test it.
I have written a demo to showcase but the actual datagrid is not open-sourced yet. It would take a while, stay tune and I will write a post when it’s ready!!
