Analytics Trends 2019 in Canada
The main focus of Rock Your Data is the Canada market. Our goal is to help Canadian companies accelerate their analytics efforts and execution of strategy.
We constantly communicate with vendors, customers, and organizations in Canada and outside of Canada and we have our own forecasts for 2019.
Before we start, we want to cover a couple of assumptions. Since we are based in the Western part of Canada, we are biased towards this Region. Some of the predictions are applicable to all industry.
The increase of interest for Cloud Analytics
There are three main cloud providers available on Market: AWS, Azure, and GCP. All of them are extremely popular in the US and there are lots of use cases available across all business verticals.
In Canada, most of the organizations are aware of them and more or less understand the benefits of the cloud. But historically, Canada moves slower than the US. There are lots of tech startups, e-commerce and so on, who already is using Cloud. In many cases, companies have a hybrid deployment.
But what is about the medium to enterprise companies? We have a huge on-premise deployment. The cost of ownership is growing. Every year solution is moving towards term “legacy” — legacy data warehouse, legacy business intelligence and so on. We think, that 2019 is a peak for the companies and they will adjust their strategy in order to reiterate their existing analytics solution and give another try to transform their organization into data-driven.
Last year, was a big year for the US, many companies explore Snowflake, Matillion, Big Query and so on in order to migrate their legacy system and get the advantage of elasticity and cost efficiency.
2019 is an awesome year for Canada to check the full potential of Cloud Analytics and deploy Proof of Concept or even migrate to the Cloud.
However, there is a big stopper. Despite the fact of the availability of cloud services, we have a problem with the availability of physical data centers in Canada. For example, AWS has a data center in the Central region and Montreal, but it could have bad latency for the West Coast. GCP is available in Montreal. Azure is available in Central and East region. In most cases, Western Canada companies are using US data centers but in some cases, it is prohibited.
As a result, Western Canada should trade performance to be aligned with Canada law or use US West data centers.
Anyway, the trend is strong and positive and we are happy to support any initiative of local companies. RYD is a partner of GCP, Azure, and AWS.
Adoption of Modern Analytics and Architecture
For us, Modern Analytics Solution consists of individual elements that are available in Cloud and do their job. For example, we have solutions for Big Data, like Elastic Map Reduce, we have solutions for streaming Analytics like Kinesis and Kafka, we have solutions for a Data Lake. Finally, we have classical Data Platforms like Snowflake, Redshift, Azure DW and BigQuery.
The benefit of this architecture, it consists of individual elements that can be scaled up and down individually. There are some other examples of modern architecture framework:
The solution above is built on AWS and using most of the components of modern Analytics Solution such as Data Warehouse, Real Time Streaming, Big Data Analytics and so on. It is secure, scalable and cost-effective.
Another example is using Azure:
Using Azure or GCP you can build absolutely the same architecture. All cloud providers have building blocks.
Based on our observation the Microsoft is historically dominated on the Western Canada market. It reminds story with Microsoft Windows that was a major OS for decades.
In most cases, organizations will choose the same cloud to provide in order to be aligned with their legacy system and skill matrix. For example, if a company was a Microsoft Shop than it will prefer Azure.
Our personal opinion on this topic, that it doesn’t matter which one cloud provider you will choose. All of them are pretty the same nowadays. Just try them all and check which one better.
Streaming Analytics
Streaming analytics isn’t new. However, the key distinction now, that you don’t need to do rocket science in order to streaming data. Cloud offers us easy integration and set of tools native, open source or 3rd party.
Business is always exciting about real-time dashboards. And now we have a good set of tools to deliver business-critical operational reporting. There is a couple of example of reference architecture:
And there is the same for the AWS:
There are a number of solutions like Apache Kafka, Confluent, Amazon Glue or Azure Data Factory as well as Google Cloud Data Flow.
2019 is a good year to try streaming analytics using modern cloud solutions.
The grow of 3rd Party tools
Despite the fact that Azure, AWS, and GCP provide all tools for data analytics. Often customers are looking for 3rd party alternative. For example, you may compare AWS Quicksigh and Tableau. Tableau is an absolute leader on the Market and Quicksigh is lightweight visualization and analytics tool.
The same story in data integration, AWS provides Glue, but it is very limited in terms of functionality. On the other hand, we have Matillion ETL that is super fast and convenient ELT tool that is available for major Cloud Data Platforms.
Finally, we want to mention Snowflake. This is a true Data Warehouse is a service that is the cheapest available solution across competitors due to unique architecture. It splits Computing and Storage.
As a result, we believe that Canadian companies are looking the best of breed solution using Cloud as a foundation and fill it with best 3rd party tools.
In order to demonstrate this, we will take our Azure architecture and replace it with 3rd party tools:
This is a good example. We replaced Big Data solution and Data Warehouse with Snowflake. We got rid of data integration complexity and replaced it with Matillion ETL. Finally, we used the best BI tool on the market. The cost of a solution is similar but provide you more flexibility and scalability.
Back to SQL
Structured Query Language was invented a long time ago. If you are in the industry for more than 10 years you might notice the rise of Hadoop with MapReduce, then the rise of NoSQL databases, later — the rise of Python and R. Every data professional was thinking about switching from traditional relational databases to the new fancy technologies. Fortunately, for us, SQL won a battle. Nowadays, you may find a SQL interface for almost any data related technology.
Dear Canadians, sleep well, SQL is here:)
Adoption of Machine Learning and Deep Learning
Deep learning starts with intense academic activity from the late 1950s to 1998. The foundation for the current era was laid in the 1980s and the 1990s with research from Yan LeCun on Convolutional Neural Networks and on LSTM by Sepp Hochreiter and Jürgen Schmidhuber. However, only recently ML/DL is started to spread across business verticals offering huge uplift for the organizations.
Cloud providers provide us extensive computing power at the fair price. Moreover, it provides managed services such as AWS SageMaker and so on.
Canadian organizations and data leaders are learned a lot about the benefits of ML and DP as well as use cases.
Finally, leading vendors like DataBricks and consulting companies like RYD are educated data leaders and help them accelerate their ML initiatives.
We believe, that 2019 is a good time to think Enterprise ML deployment or at least a good time to solve the old task using a machine algorithm.
Adoption of Visual Analytics
There are lots of Analytics and BI tools available on the market. The price range depends. Some are cheaper, some are more expensive. Most of the tools are available on Gartner Quadrant:
Based on the Canada market we can identify Legacy BI tools like Crystal Reports, Birtst, Oracle BI, Cognos, Reporting Service and Modern like Tableau, Looker, Microstrategy and Power BI.
First of all, companies are going towards modern solutions and this is a migration project. At RYD we did multiple migration projects and learned one cool thing, more than 60% of legacy reports are trash.
It is a very important step to choose the right tool. It is not a secret that Power BI is dominated. The answer is clear, it is a historical relationship with Microsoft. As a result, companies by default switch to Power BI. Customers love it, because it is cheap and remind Excel. But it has many hidden pitfalls.
We meet more and more companies, who are using Power BI but have issues and who tend to switch to other tools that are more efficient and powerful.
As a result, we believe that in 2019 companies will look for Power BI alternative as well as migrate their legacy BI.
Education and Training
With the rise of cloud, new tools and so on, it is critical to train people how to use these technologies. We can’t rely anymore on legacy courses and materials, there is a need for modern education programs.
There are a lot of available courses with a focus on modern analytics and cloud:
- GCP, AWS, Azure fundamentals
- Big Data and Data Lake best practices
- Visual Analytics
- Cloud Data Integration
- Machine Learning
- and so on
Examples are
Google Crash Course in Machine Learning
Data Engineering on Google Cloud
2019 will be a great year to pick up new skills and apply them at work. Keep learning!
Data Modeling for Cloud
In most cases, you might think that building analytics in the cloud is fast. Yes, it is relatively fast. Cloud allows us elasticity and it will handle bad design. However, in the long term, this solution will be expensive and inefficient.
The problem is obvious, data engineers may skip the design part and just load all data into the data lake of data warehouse and connect BI tool. This is a kind of MVP. But usually, this MVP will be your final solution.
In 2019, the quality of data modeling for the cloud will jump. Organizations will design data models and think about quality and scalable design that will cover organization needs.
You may look into the SQLDBM or DBT in order to manage your data models.
Data Governance, Data Quality, Data Lineage and Data Management
All these topic are crucial for successful BI/DW implementation. The absence of these procedures is part of the high rate of Analytics initiative failure. You may modernize your data platform, move to the cloud or build a brand new analytics solution, but eventually, you’ll get chaos and user will start to complain about the solution.
We believe, that with the rise of cloud analytics it is important to care about Data Governance activities and take it very seriously. In addition, most companies have to meet GDPR requirements that will force the needs of proper Data Management.
In order to help our customers with this journey, our team passed the exam for Data Management Professional and follow DAMA standards.
DAMA International is a not-for-profit, vendor-independent, global association of technical and business professionals dedicated to advancing the concepts and practices of information and data management.
This is just one framework to handle data management.
Security Bar
Another downside of speed is Security. You may move fast, however, don’t forget that Cloud provider isn’t responsible for the security of your data and application. It calls Shared Responsibility Security Model. All cloud providers and 3rd party tools provide features and techniques that will help you to follow security guidelines.
In 2019 companies who already started to use cloud will learn more about their cloud deployments and will audit their data and solutions. Moreover, cloud providers offer lots of tools and instruments for Auditing and ensuring that your data is secure enough.
The boost of Data Communities
At 2018 we saw a rise of Analytics Communities across Canada. At 2019 we are expecting even number of Analytics Meetups, User Groups, Webinars, Workshops and so on. Collaboration and knowledge share is a key driver for Analytics evolution and transformation.
RYD is committed to run Analytics meetups and user groups every month in BC and share our use cases in Cloud Analytics world.
Analytics for Public and Government Sector
We see big interest from Canada government and public sector in analytics usage and modernization of their legacy solutions. They have lots of internal barriers related to Canada Law and Security. However, they have talented employees and leaders who are supportive of innovation and modern analytics. They just move slower than the private sector and they are waiting for more Data Centers across Canada in order to cover their needs.
Their goal is to make a life of Canadian citizens better and they are using data in order to achieve this goal. They are collecting terabytes of data and searching insights. Using modern analytics solution will accelerate them.
We believe that in 2019 public and government organizations will explore the potential of migration towards modern analytics solutions. They will start small, from TOP by replacing their legacy analytics tools and replace it with powerful Analytics Platform like Tableau.
Quantum Analytics
Recently IBM introduced quantum computer. This is a huge achievement. We don’t think that we will see any quantum analytics soon but RYD is committed to delivering innovation analytics solutions for their customers and we definitely will explore opportunities for our customers in the future.
