Storing data in 2019: trends, problems, and potential solutions in Panoply’s yearly report
We at Room42 work with data on a daily basis and try not to assume things, but to see the collected information first and make reasonable conclusions based on figures. As we saw Panoplys’ yearly report on data warehousing technologies, we ran quickly to take a look at it.
Data Warehouse Trends Report has been published since 2017. In their research, specialists from Panoply interview data-engineers, scientists, and IT experts from different spheres. The most important question is how qualified workers manage their data and if their experience is satisfying. This year they have talked to as much as 800 respondents who helped define the main complications and to realize that the market needs innovations the sooner, the better. You can read the original report here.
For those who don’t have the time to read it, we’ve summarized the key insights and are ready to tell you the main takeouts. Here is what we have managed to find out:
Data warehouses are a must! (and they better be cloud-based)
Professionals are moving further and further away from on-premise data warehouses. They’re looking for a cloud that would be user-friendly and fast. According to the poll, only about a third of all respondents still stick with the in-house storages (compared to 39% a year before). The rest prefer Redshift, Azure SQL Server, BigQuery and other SaaS-solutions.
In the last three years, the most popular service Redshift has lost much of its’ market share: in 2017, 60% of all experts claimed to have used it, but in 2019 this number went down to 40%. The “Other” option in the survey is gaining popularity: its share has grown by 13% since the last reporting period.
Source: Panoply Data Warehouse Trends Report © Panoply Ltd. 2019
Current solutions are hard to use
One of the main insights that we took away from this report was that existing tools are not user-friendly at all. They’re not suitable even for the best professionals and their goals. Moreover, up to 80% of employees from smaller and larger firms describe their experience with data warehouses as “difficult” or “very difficult”.
Interestingly, the number of unsatisfied people grows year by year and doesn’t promise any positive changes.
Source: Panoply Data Warehouse Trends Report © Panoply Ltd. 2019
As we look at the respondent sample, we can observe that the problem is not in workers’ readiness to manage large amounts of data, but in the functionality of the solutions. Most of the interviewees are employed in finances, B2B, e-commerce, or state/local governments.
Source: Panoply Data Warehouse Trends Report © Panoply Ltd. 2019
As well as this, around 40% of them are working for larger businesses (up to 5000 people), and around 19% are employed by companies with 1000 to 5000 people.
Source: Panoply Data Warehouse Trends Report © Panoply Ltd. 2019
What exactly do users hate about data warehouses?
First and foremost, their complexity. This issue is common for the majority of the existing storage solutions: from Redshift to Azure SQL Server. The price and performance annoy users less frequently (from 14% to 22% of the time).
Source: Panoply Data Warehouse Trends Report © Panoply Ltd. 2019
As for the general disappointment, all of the participating data warehouses show unimpressive results: from 50% to 64% of the respondents are upset about hanging or slow queries.
Source: Panoply Data Warehouse Trends Report © Panoply Ltd. 2019
What does all of this mean?
Endless (to say the least) opportunities for modernized, simple-to-use, fast data warehouse solutions in the game. Current market leaders seem to have given up on the existing problems, and the new players haven’t entered the competition just yet. So, the road to success for fresh ideas is wide open.
Luckily, there are inexpensive services such as Differture.com which can assist in solving some of the issues in collecting, managing, and storing data.
We hope this article was helpful. Until next time!
