Big Data

Spark: Aggregating your data the fast way

This article is about when you want to aggregate some data by a key within the data, like a sql group by + aggregate function, but you want the whole row of data. It’s easy to do it the right way, but Spark provides lots of wrong ways. I’m going to go over…

Is Shapeless worth it? What is Shapeless anyway?

Getting started with generic programming in Scala, and best practices for generic programming

I’m not sure this is right. Random access busts caches (slowing reads), and more generally linear access is more capable of being optimized by preloading. This effect is even greater when data are spread across different network nodes (because network IO is relatively slow and relatively easy to saturate).