Reflections on 2017 and Predictions for 2018
We had a great year and learned a ton from the data science community, thanks in large part to our customers. 2018 looks even more exciting!
Data Science Wild Wild West
Gosh, there sure was a lot of confusion about data science in 2017. Heck, we were confused too! But, there is a bright side to it. Data Science is everywhere now! Four years ago the hype was centered around ‘big data’, that is consolidating huge amounts of data from various sources for analysis. However, organizations quickly realized that viewing historical trends were not enough. Not only did they have to view historical trends but they also have to use models to predict events on unseen data. Organizations have to react in real-time.
This stuff has been around for centuries! How likely is it that someone is going to die, or not? How likely is it that this person will pay their bills on time, or not…this list goes on. But, now we have access to huge troves of data and compute power at very cheap prices. So statisticians, data engineers, and business analysts joined forces to create models to predict the future! How likely is it that this picture is a hot dog or not a hot dog…you certainly couldn’t do that several years ago on the cheap.
2018 is the Year of Data Science Clarity
In our opinion, there won’t be major shifts in technology that will change the data science landscape, even if all of the sudden there is a breakthrough with quantum computing at room temperature. 2018 will be the year when most stakeholders within an organization will understand all basic concepts of data science and how it can impact their business. From a vendor perspective, we see the data science landscape as follows:
- Consultants and subject matter experts: these are highly skilled professionals working on training teams to obtain data science skills.
- Data munging and big data vendors: legacy ETL (Extract, Transform, and Load), big data (such as Spark and Hadoop) vendors will continue to improve their suites to help data scientists understand their data and prepare it correctly for the model development process.
- Automated Machine Learning (AutoML) solutions: open source tools and proprietary solutions will continue to improve the model selection and tuning process. This is essentially machine learning for machine learning and is a big time saver for many data science teams.
- Open Source Exploratory Data Analysis (EDA): Jupyter Notebooks keeps rocketing in popularity. Many academic institutions are using Jupyter Notebooks to train computer scientists, data scientists, software engineers, business analysts, physicists…Jupyter is everywhere!
- Deployment Solutions: many options have appeared to allow data scientists and software developers to deploy their trained models using resilient cloud compute services.
- Deep Learning Frameworks: it’s great to be involved with neural networks. In case you haven’t noticed there is an arms race with deep learning frameworks and we are the beneficiaries! TensorFlow, MXNet, CNTK, Lasagne, PyTorch… are all fantastic en love creating models with these solutions.
- Canned Solutions: examples include AWS Polly or Google Image API. These are services offered in the cloud with well-documented APIs so developers can quickly integrate Machine Learning and Deep Learning services into their applications. In time, more specialized options will become available.
- R and Python: have you seen the latest programming language popularity reports? Enough said.
- We need to give Docker containers and Kubernetes an honorable mention. These container solutions have, in our opinion, helped tremendously to build consistent data science environments.
Organizations, individuals, and vendors will all find their place and focus on providing amazing solutions. Everyone will become more versed with data science terms such as “classification and regression algorithms”, “bayesian optimizations”, and “recurrent neural networks”. Our conversations at the dinner table will revolve around how a new, cool app hit the market and how AI helped it crush its competitors. Most importantly, decision-makers will understand how to filter through the fluff and select good options for their organizations.
As for us, we are just thrilled to be a part of this amazing journey!
Would you like to join us? Sign in today, no credit card required. We would love your feedback!
