SQL is Dead, Hail to Flux
Have you ever wondered what’s the worst nightmare for a software engineer? Is it untested spaghetti code? Is it.. overtested structured code, when touching a piece of it feels like hammering a work of art? None of that.
The worst nightmare for a software engineer is to miss trends. We all have references of trends we missed, and we can’t help but to feel terrible at night about it.
Why didn’t I invest in cryptocurrencies when it was worth only one dollar? Did I miss the Javascript wave? Am I late to the party? This time it won’t happen. By means of this catchy title, I wanted to introduce you to Flux : a lightweight, blazing fast scripting language that makes it super easy to work with data.
I) Features
Flux does not aspire to be a SQL 2.0. Not at all. Flux is a brand new language that reconnects modern concepts such as open-sourcing, modularity, expressiveness and shareability with the scripting world. It is built in Go for enhanced performance.
Flux is built for :
- Expressiveness & Readability
Right above is an example of a Flux query. The major asset of Flux is that queries speak for themselves, they are expressive. Instructions are piped together using a pipe operator, and each step applies a very specific function to our data. No more nested queries, no more inexpressive SQL mayhem. You get exactly where the data is taken from with the from clause, the range of the data we operate on, and all the subsequent transformations we apply to it.
When it comes to coding, naming functions, and even building architectures, systems and codebases have to reflect the intent. Code is read by humans, and without a proper understanding of what is written, it is only a question of time before your system becomes unusable or unmaintable.
There’s always a right middle in designing programming languages between purity and usability. Some languages have the reputation of being very pure, yet they are not largely adopted because they are not usable or scalable to the systems we want to build. Infamous examples of such programming languages are Lisp and Haskell. Extremely pure languages, sounds good, doesn’t work.
On the other hand, Flux keeps a right middle between those two concepts. You will re-encounter functional aspects such as anonymous functions, and function composability. Yet you are left with a very usable programming language, that does what it says it does.
- Make everyone a data programmer
Simplicity often rhymes with popularity, and Flux aims to be as popular as possible. Flux is not reserved to a subset of elite programmers or data specialists that have the top knowledge of machine learning algorithms. Flux wants to make data manipulation easy and reinforces it with simple and atomic functions. With this framework, anyone can manipulate data. Anyone can play around and build models that suit their needs.
Flux even comes integrated with a tool called Chronograf, a data visualization tool for programmers. How cool is that? You type your query, and you get a direct visual representation of what you just typed. SQL beware.
- Contribuable
This is probably what excites me the most. Flux is a language designed to be contribuable. By building simple atomic functions, and by allowing developers to build their own functions, anyone can share functions with the community.
Imagine a world where you don’t have to write the same functions again and again, because developers have already written it for you. They even tested the function for you.
Isn’t it developer productivity we are talking about here?
The major issue with SQL is that everyone has its own little way to perform an operation, its own little tricks. A package management system would ensure that functions are only written one time and tested in one place.
Developer productivity is at stake here. And if we can find a way to ease writing and testing data processing functions, the data science world would benefit from it a lot.
We have seen it with React. React created the concept of component and component reusability. Now React is one of the largest ecosystems in the front end industry with hundreds if not thousands of components created everyday. Want a video player? Got a component for that. Want a fancy dropdown that sings Merry Christmas when you open it? Got it.
Flux could be the same. Want a function that computes the moving average on your data? We got a function for that.
- Testability
Unit testing in the data scripting world isn’t a trivial problem. SQL unit testing suffer from being heavily correlated to the data structure behind, which is not the case with a functional scripting language such as Flux. Even if Flux functions directly manipulate and transform data, they are not as directly connected to the data structure behind, which is a very reliable asset.
Flux functions can be tested in complete isolation from the outside world, thus isolated from the data structures it is applied into.
Flux wants to reinforce flexibility, reusability and readability. It bets its whole existence on the fact that scripting languages are far more effective to manipulate and transform data than query languages. I think it is a very valid approach for our times. With the rise of machine learning and artificial intelligence, working with data has become a very particular skill set on its own. Scripting languages have really become first-class choices for data scientists. And Flux is not an exception.
But Flux takes another bet : the emergence of new languages is made possible by increasing the developer productivity.
Now, how do you think developer productivity could be impacted if you went from this :
To this :
III) Interoperability
So.. what’s interoperability and why is it so important for a scripting language such as Flux?
If you read my previous article, I explained why on an architectural level, you might have to use integration patterns in order to build systems that can communicate with each other in a reliable way. Now Flux does not claim to be an ESB, yet it introduces a lot of built-in plugins that let systems and frameworks communicate with each other.
The schema above describes the flux workflow. Flux takes its data from a set of inputs (InfluxDB, Kafka..), transforms it and then sends it to an output being wherever you want to store your data. This is why Flux is so great : it is not tied to Influx. You can take your data from wherever you want, and it’s a big asset for the data scripting world. No ESB needed.
Again, isn’t that developer productivity at its finest?
IV) Contribute & Experiment
A new trend is nothing without us, developers. It is only the early stages, but I do believe that Flux has some very bright days ahead of it. Remember the Rogers Curve? This is where Flux stands.
Jokes aside, I really encourage any curious and interested developers to jump the train. To test Influx and maybe contribute to its development : it is completely open source. InfluxData made a lot of tutorials on how to run its TICK stack, and it is available for every operating system.
On the other hand, here’s a complete talk by Paul Dix, CEO of InfluxData, brilliantly presenting a keynote about Flux. He goes even more in depth and explains some of the technical aspects of Flux that are not described here.
Flux is available here :
Have you already tested Flux? If so, what’s your opinion on it?
Feel free to comment on this article, clap it if you like it, and join me on my social networks if you want to keep in touch. Have fun developers!
