Big Data, Functional Programming, and Developers

Stack Overflow recently released their 2016 survey, and now we have it; a comprehensive and detailed look into current trends in the developing world. Here are some key points we found interesting at Longford & Co.


“Without big data analytics, companies are blind and deaf, wandering out onto the web like a deer on a freeway” — Geoffrey Moore

Big Data, Big Deal

Big data continues on its track of being increasingly utilized by companies to gain insight into better business decisions. Today, the ideal developer has a thorough understanding of the technologies that best manage — in analyzing and storing — huge amounts of data. Functional programming languages, in supporting parallelism, address the need to scale the increased requirements of modern applications, from storage to processing. This trend is mimicked in the survey, with F#, Clojure, and Scala all falling within the top 10 bracket of top paying tech. Functional programming, and the ability to lock in large data sets as immutable, provides assurance in dealing with large amounts of information. In scaling down complexity of programs, developers are able to decrease errors and side effects, a selling point for the companies (and their massive data sets) who are seeking their services.


Machine Learning

Though human insight is still needed to pick up on the nuances of big data, the more that this data is presented as accurate and complete into the machine learning programs, the faster this need will be eradicated. Just check out companies like Spare5, who combine crowd-sourcing with machine learning to organize and complete unstructured data sets. While they still may need the human touch to pick up on connections the algorithms may miss out on, the model will continue to improve and accurate associations will be learned. Eventually, it would seem, the goal- or consequence- would be to eradicate the need for this human element, which begs the question; Where is the role of data scientist headed?


Where does the developer go from here?

The heightened popularity of functional languages is very much a response to the more fluid and expanding role of developers. The developer is now facing bigger problems and bigger data models. Pair that with the DevOps movement, and you are left with what is essentially an operations-centric model applied through development. In order to maintain this dynamic, developers need to be well versed in technologies that support heavy concurrency and automation.

Those who can help turn big data into business insight will stand out to companies. This move to establish better business intelligence and analytics products is translated in the need for developers well-versed in technologies such as Clojure and Scala. Developers who are in a position to help create solutions that aggregate big data into better business decisions will continue to be invaluable. For those who are not, building skills with technologies that analyze, capture, and act on data as it arrives will serve you well.

Comments? Tweet @longford_and_co. Let’s get talking.