Towards efficient analytics
Abstract
Today everyone knows that data is one of the most important assets of any organization from a small startup company till enterprise. There are lots of different sources of data available nowadays such as ERP systems, Open Data, Online data, Transactional data and many other types of data that could be useful and can bring value. Despite the fact that most of the companies have Business Intelligence teams or just analysts who have enough skills to work with such a tremendous volume and variety of data, these teams and analysts could be a potential bottleneck for the organization, especially in the modern world with the fast changing environment.
Most of the companies think about BI teams as it is a part of the IT department and just create tickets for them. As a result, BI team simply accumulates lots of requests from the business and struggles with them. As a result, business users are disappointed because they have to wait or try to solve their question themselves.
Solution for this problem isn’t new. It is called Self-Service BI. However, it is difficult to implement because it meets lots of resistance from business teams as well as IT. In this article, we will look at how we could build Self-Service BI in the organization based on successful experience of multiple projects.
Current State of BI
Nowadays, I don’t know any companies who don’t generate reports and use data for the decisions. Companies can use free open source technologies, expensive enterprise analytics software. Some of them prefer to use spreadsheets for the reports. At the same time, they are collecting data into a data warehouse, also using various technologies. Let’s look at the traditional company, who implemented a data warehouse and acquired BI software. This company created a special team or department for the managing data warehouse and BI as well as providing reports to the business users. Usually, this team is a part of IT. As a result, business users think about this team as a kind of help-desk for data requests. They start to generate requests for the specific reports or for the adding of new pieces of data into the data warehouse or simply about changing the existing logic.
BI team works hard. It has data integration developers, who load data and support this process, data warehouse developers, who support data warehouse and create new tables and other database objects, business intelligence developers, who administrate analytics software and develop reports, analysts, who collecting requirements from the business and create task for the BI team, and finally, business intelligence manager, who manage the team and negotiate with the business.
Time is going and the number of requests are significantly growing. BI developers should develop more reports, data integration development should add new data sources and etc. Business users are becoming angry with the BI team and concerned about long terms and long queues. In the modern world, companies should move really fast and they can’t wait long for data in order to make decisions. As a result, this type of BI team has failed. It is just too slow to support many business users requirements in a fast changing environment.
It is time to look at another way of extracting value from data by setting a self-service analytics environment in your organization and changing the existing approach for the BI where business users think about this as a part of IT data help-desk.
Self Service BI
We can find many definitions of Self-Service BI from various experts or BI companies. For example, this one from Gartner:
Self-service business intelligence is defined here as end users designing and deploying their own reports and analyses within an approved and supported architecture and tools portfolio.
In other words, Self Service BI is a way of interacting users with data with help of analytics and visualization tools in the environment where the user has good knowledge of the analytics toolkit. However, this definition missed very important peace — knowledge of data. It is impossible to answer any business question, if we don’t know what our data is and how we can map our business process with the collected data.
There is a long journey and requires significant efforts from the BI and management teams.
Let’s look at the existing problems of the modern business user who has to report half of his work time. Every day, employees need data to understand the current state of the business in order to make decisions and grow their business. Organizations with Self Service BI can do more and spend less time.
Maturity model of organization
Before we dive deep in the process of the setting self service BI in your organization, let’s look at the analytics maturity model of the organization on pic.1.
In this example, there are 4 stages. First one is “Scattered Reporting”. At this stage, the organization doesn’t have a centralized data warehouse. It uses internal Online Transactional Processing Systems (OLTP) for the reporting purpose by dumping data into spreadsheets. All reporting is based on ad-hoc SQL queries and spreadsheets. This could be painful for the business users and organization in general.
Solution for this chaos is the corporate data warehouse. I call this stage “Centralize BI Solution”. Usually during this step the IT department builds BI team and fills the roles in order to choose and implement BI solutions. This solution consists of 3 main elements, such database, data integration software (ETL) and BI software.
After a couple months of hard work, sometimes years, BI solution is ready to use. It really depends on what methodology was used during the process of the implementations. Some organizations like waterfall and other scrums. Personally, I prefer scrum, because in this case the solution can start to bring value much faster and it is easy to update requirements during the retrospective phase.
Time is going and the volume of data is growing as well as the number of data sources. There are performance problems appearing or just new requirements related to analyzing digital data. As a result, companies start to look at big data technologies such as Hadoop or similar in order to store and process huge volumes of variety data. This is time for the “Big Data Analytics” stage.
Finally, the company wants to process data in real time as well as make proactive decisions via mobile devices and put hands on the data pulse.It is time for the last stage “Real-time Mobile Business Insight”.
It isn’t really important for the sequence of these steps as well as the technologies that are used for each step. The main thing here, that all organizations are growing in the analytics field during their lifetime. It is obvious that for this to grow, companies need significant resources as well as expertise of BI team. But if your team is struggling with the report preparations and doing lots of business requests, there is no time and resource for the analytic growth. As a result, companies are losing to their competitors, who are using data and grow fast.
As you might guess, the solution for this problem is Self Service BI. I am going to share from my experience the things that should be done in order to set Self Service BI and push business users to dive in the data and answer most of their business questions starting from the assessment of your organization.
Assessment of your organization
One of the best ways to listen to your users is to conduct a survey. Even better to run a survey once per quarter using the same list of the questions in order to measure success of changes aimed at the better usage of BI solutions and available data. There are lots of various questions you might get for the survey, but they should be clear and transparent for the business users.
Surveys play a significant role in building relationships between BI teams and the business teams because it allows to collect feedback from the business users itself and and focus on solving problems and eliminating any barriers for the self service BI.
First of all we should choose methodology for the survey measuring. There are two of them the most popular such as Internal Customer Satisfaction Index (ICSI) and Net Promoter Score (NPS). It is up to you, what do you like to use. Personally, I prefer ICSI, because it has grades from 1 to 5 and it is easy to measure results.
I would like to show you an example of an effective survey that works very well and there are just ten questions. There is one important thing with these surveys — we should be honest and don’t try to use questions that will demonstrate BI from the positive side only because it would be just a waste of time and the result of this survey will be meaningless.
Before you start, make sure that you have one BI@ email group for the communication with business users. This group should keep all users that are related to the reporting and use data for the decision making.
There is an example of an effective BI survey in the Table 1
Our main focus is questions 4–11, which we actually measure. When a survey is done, we can grab raw data, put it into the BI tool and build a nice visualization that can tell us a lot about existing problems and blockers. There is a one of chart of sample results of an survey on the picture 2.
Looking at this data we quickly understand what should be our main focus in the nearest time. Our intention should be increasing all low marks. Especially those that are important for the self service BI such as “Business Glossary” (Availability of information about the available data / calculation methodology of metrics) and “Speed of reports development”. Moreover, we should know how many people are actually using BI solutions. I bet that many of them prefer to use BI tools only for dumping data from the report to the spreadsheet or ask BI team to build SQL queries and use it for getting raw data for the future report in Excel.
How to start
After the survey we should eliminate barriers for the business users by providing good documentation about the semantic layer or business metadata in the BI solution. We should focus on providing the definitions of the metric, their calculations logic and data source. For example, in reporting business user drag and drop Net Revenue. Despite the fact that this metric is very common and easy to understand, various teams in the organization have different definitions for one metric. As a result, they could report different results to the manager. That’s why it is very important to create a semantic layer of BI solution as a solid single point of truth with detailed documentation. In order to answer all possible questions of business users and eliminate any barriers for the self service.
The next step is to find a good example of the self service BI at your organization. Usually, it is taught to start for the employees and they are resisting changes because they are stressful for them. But some of users are happy to change and quickly figure out how to work independently. AS a result, they can solve their problem themself by building a dashboard or creating a data mart. It is important to organize support of business users by setting BI open hours, BI weekly meetings or data breakfasts. Data breakfast — it is the begin of the work day when BI member meet business users, grab their breakfast and solve business problem. When first business problem solved by business user, it is time to share this success and learn about it, what was good and what was bad in order to improve this process in the future.
Finally, BI team should really focus on providing quality training for the business users by teaching tools, business metrics, use cases and data.
Train employees
There are lots of ways of delivering knowledge to the users. However, I found that weekly live training doesn’t work at all, because business users haven’t enough time to attend them. That’s why building video content with the exercises is a good choice. Another useful tip is to think about branding of the learning process. For example, in the picture 3I have an example of the Lamoda BI Academy (Lamoda is a huge online retailer).
Moreover, to make it fun, I used software that allows me to build cartoons. On picture 4 there are two main heroes from the video lessons that will teach you about the BI solution.
During the process of the course design, it is important to split it into two streams Technical and Business. Technical course will focus on the technology side of the BI such as BI Tool, SQL, database, ETL and etc in order to teach users technical skills which are required for daily self service. Business stream will teach about the data, metrics and especially use cases in order to get accurate metrics.
After education, there is a small exam or certification of the business user by BI team. Everyone who passed the exam will get a present, for example a T-shirt on picture 5.
For sure, business users don’t have enough time to learn new tools, that’s why it is important to convince business management to push their employees to learn new skills and data in order to be more proactive, faster and generate accurate reports for the right and quick decision. Moreover, it is good practice don’t buy a license or don’t provide access to the person till he gets enough skills.
Finally, we can try to convince HR to use these skills as a part of career development in the organization.
Pick up a BI Champion
Despite the fact that people will learn and start to work for themselves and benefit from the self service BI. It is important to find for each team a person, who has the deepest knowledge of the tools and data. This person calls the BI Champion and he should be a bridge between the business team and BI team. In addition, this person should spend enough time learning tools and data. This is a good practice to obtain a couple of certificates of BI tools or data warehouse solutions.
Add Transparency
All these things will give enough time to the BI team to focus on the growing of analytics expertise, implement new use cases and move towards the maturity mode. However, it is important to get transparency on the process of self service BI in the organization.
One of the drawbacks of the Self Service BI is that everyone starts work very independently, and it is difficult to understand what’s going on. There is a simple solution for this case, that works very well. I call this BI Kanban. This is very popular in the Software Development work and also with Scrum. The idea of a BI Kanban, that everyone should put his ticket on the board that has several columns such as “To Do”, “In Progress”, “In Test” and “Done”. There is an example on picture 6.
The tickets should be broken by the business team and should be ordered by the priority from top to bottom where tickets on top have the highest priority. It gives opportunity for the BI team to review work of the people and help with the highest priority tickets.
Shared Knowledge
It is extremely important to share knowledge across the teams….
User groups
Getting everything together with BI Portal
Finally, we need a place where we can put all important information for the Self Service BI users, where they can access business glossary, use cases, technical and business trainings, discuss the topics on the
References
