Hard lessons from Self-Service Business Intelligence

By Kevin D and Steven Ingram

At Wave, we promote an analytically informed culture that uses data insights as key inputs for making decisions. Performance tracking, hypotheses testing, building predictive models, and mining for deep business insights are all critical to the success of Wave in moving fast in an intelligent way.

In a series of posts, we will share our success and struggles in moving through the stages of the analytic maturity curve (a concept written by Tom Davenport and Jean Harris in ‘Competing on Analytics’), progressing beyond localized analytics culture and embedding that culture into the fabric of our DNA. Our story is continuously being written as technology and the industry of data and analytics continues to evolve at an incredible pace.

About Wave

Wave is an award-winning tech company that provides financial services and software to small business owners. More than 3 million business owners, from 200+ countries around the world, have used Wave to track in excess of $240 billion in income and expenses and deliver more than $20 billion in invoices per year.

The Need for Business Intelligence

Startups have a particular way of beginning their journey toward a proper BI strategy. Proper BI isn’t often deployed until a startup reaches a certain scale. Instead, the production of Reports and Analytics is initially performed by the product Developers or Engineers. They’re the ones who built the product (including its databases), so they are often the only ones capable of accessing the data. Engineers serve the organization’s data requests in the form of ‘data dumps’, which are then manipulated in Google Sheets / Excel on an ad hoc basis. As the organization matures, reporting requests accelerate and multiply to a point where report generation, iteration, and maintenance start to become too complex of a process and take far too long for Developers to operate effectively. Often, this is the stage that triggers a Startup’s foray into the world of BI.

The cornerstone of developing an analytically driven culture across an organization is the BI software you choose. It becomes the platform that democratizes access to your data and the place where most data work happens; not democratizing in the “Big Company” way (aka — “Here’s your Dashboard of Data!”), but really, truly allowing users to be flexible with a BI deployment. At Wave, we chose MicroStrategy to be our cornerstone in March of 2017. After one year, we wanted to share our story so far, and what we’ve learned.

MicroStrategy? Who’s that?

In the startup community, when you think of BI more familiar names are the likes of Tableau, Looker, Chartio and a plethora of other “modern” BI platforms. MicroStrategy is a “traditional” enterprise BI platform more often associated with larger enterprises and in more traditional industries such as manufacturing, banking, and insurance.

MicroStrategy definitely doesn’t have a “new school” vibe. So, how does Wave, a company that’s probably eclipsing technical talent anywhere else in the world, a company that can and does stay very close to leading edge for a great portion of what we do, choose a traditional tool like MicroStrategy?

Our selection method, loosely driven from Kepner-Tregoe Decision Analysis combined with Internal PoC’s evaluated both traditional and modern BI vendors including Pentaho, SAS, Sisense, Tableau & Looker. At the time, MicroStrategy ranked the highest for the things that matter to a business that has the analytic needs of a small bank.

Wave:

Has multiple, disparate user groups that need to collaborate together but not always see each other’s stuff (we take security and privacy of data very seriously)

• We were able to set up a security framework while maintaining the ability to collaborate across different departments and the platform allowed us to lock down features and functionality all the way down to the column and row level on physical tables.

Has consumers of data and insights that range from novice to expert in analytics

• We knew we couldn’t train the entire organization to become ‘data analysts’ so we were able to create templated dashboards that serviced the novice users and allowed the more advanced users a framework to develop on top of.

Has the need for different delivery vehicles, not just Dashboards

• Not everything needs to be built in fancy dashboards. Microstrategy provides a few options in creating simple reports (a single table) and higher level of flexibility with ‘documents’ (a highly customizable report built off multiple datasets.

Needs to audit itself and report back on query activity

• This is something you often don’t see with modern BI applications. We have set up detailed reports on the performance and utilization of the utility by users which would better help us optimize and maintain the platform.

Has lots of connections to different data sources and is expanding all the time

• Out of the box, many of the different connectors were available for user’s to self-serve outside of the data warehouse.

Uses Excel and will continue to use MS office products (Google Sheets is not an Excel substitute!)

• Integrates with MS office (Windows only, does not have Office support for Mac), allows users to import and export data to/from Excel. It does not only rely on database technologies, which was the biggest drawback for many of the modern BI platforms.

Even if it was completely against the grain of cool, MicroStrategy fit our criteria most closely (don’t read perfectly here). When working, MicroStrategy does a pretty great job.

In accepting MicroStrategy, however, we understood that it wasn’t going to be startup friendly. Here’s what that meant:

At the time we rolled out, there was no easy and inexpensive cloud deployment option. It was on us — lots of configuration management and engagement with our OpsEng Team would be required to get MicroStrategy going.

A great deal of learning is necessary (read: library of congress) to use MicroStrategy, from both the user and Administrative ends.

• If you’ve had experience with traditional enterprise systems like SAP and Oracle, you’ll know they require specialists to implement, especially if you’re new. The same holds true for MicroStrategy. We ended up leveraging external partners GenSquared to accelerate us through the learning curve and support us in the initial installation.

Unlike some of its competitors, there’s a great deal of setup to learn:

• To take full advantage of MicroStrategy we structured our database schema to meet the design requirements of the “MicroStrategy Object Model” — the logical data model that guides MicroStrategy on how to create the SQL in order to join tables, create grouping attributes, aggregate metrics, defines column relationships, etc.
• It’s not cut and dry to make “The Model” — there’s a lot of work in the ‘plumbing’ to make it easy for business user’s to self-serve.

MicroStrategy is not optimized for user experience

• UX is something Wave prides itself on as we design our application in a complex domain of accounting, payroll and sales, and we make it easy for customers to understand and engage. The most common feedback from our users was that the tool is “overwhelming” and that the Platform was difficult to learn, navigate and ultimately adopt.

Once set up, we found that companies that normally run this product had a fleet of people to operate it. We’d have to do what they did with a shoestring budget and a handful of people. While we knew it was going to be a lot of work we believed with the right design decisions we could create a VENN intersection of Wave and an awesome BI deployment with MicroStrategy.

Launching BI into Wave

Who did we deploy to? In a Company of ~175 People, almost half wound up getting a license for MicroStrategy. The audience was broad — executives, software delivery, marketing, finance and other business operation teams. Within these teams there would also be a broad spectrum of user types from analysts to consumers.

This was a big deal. This was the tool across Wave that all teams would be using to get data and insights from.

We took our time and made sure that the setup for MicroStrategy was done, in our way, correctly. We deployed their latest release at the time (10.6) and immediately started shaving dozens of hours off of manual and repetitive analytics processes each month. We established role based security, so users wouldn’t see stuff they weren’t supposed to. We were able to start auditing our BI tool for the first time ever, as we had metadata. We optimized for performance by tuning our data warehouse and ETL processes to ensure that MicroStrategy was speedy. Users were now able to get insights quickly from our data with nothing more than a few clicks of the mouse (no coding!) and an understanding of the Business. Our users were, themselves, going out to get more training from MicroStrategy because they saw the power in what it could do. With all of the work that we were doing in conjunction with people in the Business, we were making progress. Great progress, in fact it was smooth sailing…

Hitting Some Rough Waters

Of course, there were some small encumbrances that we wanted to get past. Much effort was spent on the constant tweaking and refining of the configurations in order to achieve what we had planned to set up. There is an incredible amount of settings and poor documentation in understanding what they do and how they work. We encountered a number of errors that were incredibly frustrating to debug. Including ambiguous error messages with no details at all, some even stopping our deployment from time to time. We had to spin up a Windows machine in order to use the administration and developer client (we are a Linux shop).

In October 2017 we attended MicroStrategy’s symposium in Toronto where they announced the launch of a new version, 10.9, that included the release of some exciting new features. ‘Dossier’ was a redesign of their ‘dashboard’ feature which became more akin to Tableau’s Stories. ‘Libraries’ was also announced, which is a new collaboration feature. These additions and changes were incredibly exciting for us and where modern BI platforms exceeded. This release seemed to be MicroStrategy’s move into bridging the gap between traditional and modern BI, and we were excited to upgrade. As a startup we can move fast — where larger organizations may take months of planning and implementation to upgrade a system, we were able to move within weeks. In late November of 2017, we decided to upgrade MicroStrategy to version 10.10.

The Iceberg

The upgrade went perfectly. We were operating at 10.10 — the latest and greatest. Pretty Dossiers (dashboards). Lots of “ooo’s” and “aah’s”. Then, one of our Data Engineers noticed something peculiar.

Unbeknownst to almost every customer of MicroStrategy on the planet (certainly to Wave), that the “Connect Live” feature, which allows a user to leverage a properly built data model to dynamically query data directly from the data warehouse within a dashboard was removed. This feature was the backbone of our implementation and infrastructure design decisions that leveraged a powerful data warehouse (Redshift) and building templated dashboards that would service the broad spectrum of our users. These were strategic implementation decisions to mitigate our resource constraints and lack of tool specialization to deploy an enterprise grade BI platform.

The deprecation of the ‘Connect Live’ feature invalidated our BI deployment strategy using MicroStrategy.

This was a massive, massive setback for Wave. Why?

At any organization self-service BI adoption is a challenging enough task. While we were quite successful with the launch of MicroStrategy, we had to train the organization to behave a certain way to leverage the “Connect Live” feature. So when the feature was taken away we had to tell our users that they could no longer do the thing we told them to do.

After the upgrade we saw a lot of unexplainable behaviours leading to inaccurate numbers in the dashboards that were built off the ‘Connect Live’ feature when simply adding a new attribute. At this point we lost all trust in the numbers being produced from legacy dashboards in MicroStrategy.

What’s worse than making a decision with no data? Wrong data.

We had to rebuild all our dashboards the ‘old school way’ by writing SQL directly to the data warehouse creating predefined reporting cubes. This would invalidate the need for the logical object model and limit the ability of our users to discover and explore the data.

This would also invalidate our decision to leverage our data warehouse for performance as it now heavily relied on the sizing of the MicroStrategy iServer (the main component of MicroStrategy).

• A deployment that relies on ‘data cubes’ requires a server with high memory storage, especially as you have half your organization on the platform. These kinds of servers can get very expensive.

So in summary, in order to use the tool with any of the work we’d already done, we’d need to re-do that work and retrain all our users on how to ‘self-serve’. The MicroStrategy performance seriously degraded as everything was now being stored in memory (our CEO even called us on the degraded performance — talk about egg on our face). Months of work became immediately devalued.

Our frustration wasn’t that they removed a feature — companies deprecate features all the time. MicroStrategy didn’t communicate the removal of this feature to any of their Customers or Partners either verbally or through upgrade notes, or even to most of their internal team. If we had understood that the feature would be deprecated, we would never have proceeded with the upgrade.

Where We Are Today

By removing one feature, MicroStrategy very much devalued our BI deployment and put us off track on our overall BI strategy. What to do?

We had already made the decision to go down to the “MicroStrategy World” conference — the timing of this happened to coincide with the escalating severity of the problem we were now faced with. MicroStrategy offered to meet and listen to our concerns, they further offered to fix the problem by offering support to “downgrade” our system to the latest version with the ‘Connect Live’ feature, and then later upgrading to a version where the feature would be re-introduced.

Sounds pretty good, right? Consider:

• Any re-work we’d already done would be lost
• All of the users net new work would need to be re-done
• Hours and hours of testing would still be needed to make sure we had the correct functionality and everything was performing like it was supposed to
• We’d have to inject work and expense into other teams to get a downgrade done
• We’d have to incur additional expenses of servers, configuration time and personnel dedicated to the task
• We’d make no progress during the downgrade, which we estimated could last weeks
• We’d have to repeat more of the same for an upgrade down the road

After some thought, we decided that downgrading only to upgrade later wasn’t going to really work for us. We were in a place where we had to choose between rework and different rework. Where some companies have people and time to burn for this kind of thing, Wave doesn’t.

Unfortunately, it would seem the lesser of two evils was to re-design our implementation to work with the ‘new’ MicroStrategy and re-train users rather than re-building the system in a lower version.

Going forward

We’re incredibly disappointed about what’s happened. Everyone at MicroStrategy has been very nice, but we’re not sure they understand what this actually meant to us. We were so hoping to tell a success story — “Old School BI rattles Startup World”. We still feel we made the right decision in the tool, but would never have seen this coming. Even with all of the awesome we’ve brought to our Business with proper BI, we don’t feel like we’re set up for long term success with MicroStrategy. It is still uncertain whether they will be adding the feature back into their platform in a future release.

We have to stick with MicroStrategy for a while — that’s the problem with a BI Tool, it’s like a Canadian Cell phone contract — once you’re embedded in one it takes time or a lot of money (or both) to migrate off.

The great thing about a Startup is that it can be flexible and tactical. We’re prone to moving on things quickly. Wave has options.

A Few Lessons

Even though we can’t go back, we know we can move forward. Here’s some stuff we’d like to share.

For other startups

• To empower yourself, make sure you factor in a “tool agnostic” approach to modelling your data for BI wherever possible — if you can, try not to architect data to work with a tool.
• BI Tools are sticky — don’t buy one if you don’t do your research, and even then, know it won’t be perfect after you roll out.
• Sometimes stuff happens with BI Vendors that’s out of your control. What you do when it does will help determine success or failure of your BI Environment.
• Don’t EVER upgrade without a test environment. If you think it costs a lot to run another system, just wait until you see what happens when something key is gone. Find out the evils of the thing that you are going to first.

For Product Managers of BI

• Don’t yank key features. Full stop.
• If you’re going to yank key features, tell people that you’re going to do that. Make it glaringly clear.
• Understand your outliers — too often decisions are made for the majority. What happens to the minority when you change your product?
• If you have it in your head to simply chase competition or the Gartner Report, you may never succeed in having the best product you can. To quote Simon Sinek — know your “Why”.

For the Product Engineers of BI

• With the increasing rise of high performance databases and multi-parallel processing platforms — understand how your customers are using your BI platform in their overall architecture. BI is only one piece of the wider infrastructure design decision.
• Understand that Customers are different — some of us prefer push down code and design around using it. We WANT to use our Data Sources, as they are tuned to be the most powerful place that we do work.
• Prove to your Customers that you understand their pain — use the system the way they would, not as Engineers, but as Analysts. Find the little things that you might not consider important.