Rapid App Development — Ship Better Analytics Products Faster
COVID-19 has pushed companies to re-evaluate their traditional ways of doing things and as a result, led to innovation in the reporting and analytics space. There is actually a website tracking all COVID innovations. At the onset of COVID-19 in March 2020, we had all hands on deck to assess risk within our supply chain and report out supply surety. The ask was to build a solution around the surety program and push the program to achieve months of maturity in matter of days. As my previous article notes, our Data, Solutions & Analytics (DSnA) team is functionally aligned with a dedicated BI and software development teams. These two teams had all the potential and skillset to ship a product with all desired features but not in the timeline needed (unless we dropped a lot of other important initiatives). This led us to think — are we missing anything in our toolbox that would support this need?
At the same time many COVID related web apps and trackers were collectively being admired for their elegance and impactful delivery of information. The team discussed how so many brilliant COVID related products were shipped to the market so quickly at the start of the pandemic. How was this happening? Were teams just throwing resources at these products to ship them fast or they had other solutions available to make that happen as well?
With these thoughts in mind, we set out to find answer to this simple question — is there any framework, product or solution out there to allow us to ship elegant and comprehensive applications faster without spinning up full software development resources?
The Analytics Solutions Landscape
Upon some research, the following spectrum of tools & frameworks for analytics development emerged quickly.
- At one end of the spectrum are traditional BI development tools. These are primarily drag & drop development tools with some scripting to write formulas, prepare data & embed business logic. These tools are less complex but at the same time offer less freedom of development. Capabilities are restricted to what the tool allows/offers. These are also marketed as self-service tools. Examples include Microsoft Power BI, Tableau, QlikView and other similar solutions.
- On the other end of the spectrum is full-stack (front & back end) web application development. This is coding intensive work and needs highly skilled developers to execute. Popular JavaScript or Java derivative languages (e.g. TypeScript) based front-end development frameworks like Angular, Vue & React are popular. For back-end, there are JavaScript based options like Node.js and other languages like C++, Python, Ruby etc. are also used.
- Between these two ends of the spectrum lies an option that we were looking for. A low-code framework that allows our BI Developers to up-skill and our data scientists to leverage their existing skillset to develop and deploy applications relatively faster. Python, R and Julia are typically the languages of choice in this space and there are foundational frameworks like Django and Flask but then there are options out there that sometimes leverage these frameworks and make life easier for the App developer. Examples include Plotly Dash (basically a flask based application framework), Bokeh, Panel, R Shiny etc.
These low-code frameworks allow the developer to stay in one language and let the libraries do the translation to HTML, CSS and JavaScript. All aforementioned frameworks are open source, but some also offer an enterprise version with a software solution including an IDE, deployment engine and enterprise-only libraries (making it even more low-code) among other features. In our proof of concept for Plotly Dash, I was able to build and deploy a simple app (couple of basic graphs with a filter) with a few lines of python code in less than 15 minutes. Now that was intriguing!
Why These Frameworks?
Compared to reports & dashboards built in Power BI or Tableau, we found that a bit of extra effort to develop these applications came with the following benefits.
Ease of Developing Interactivity
Typical BI reports & dashboards have one-way flow of information (from the report writer to the user). The opposite flow is typically not available without integrating with additional solutions. For example, the Microsoft Power platform leveraging Power BI, Power Automate & Power Apps together can be used to build two-way interactivity. But you must develop in 3 tools, work on integrating with databases to handle flow of data and figure out how to keep the data updated in real-time or near-real-time (if that is the desired use case). Similarly, presenting ML models to users in a visual and interactive way is not something these tools can handle easily.
On the other hand, a web app developed in the low-code frameworks can have all levels of interactivity and data management built in. Yes, it is more work than developing a BI dashboard but it offers more flexibility and for the right use case, that may be the best solution.
Visualization Options
If you have developed in Power BI, Tableau or similar tools, you know that the types of visualizations you can incorporate in your product are limited. You use what’s allowed in the tool and anything fancier is either not possible or has to be coded up in Python or R (Power BI allows to run Python/R scripts). Complex ML models, their parameterization and workflows can’t be visualized in these tools. Whereas for an app developed fully in Python or R, you have the full universe of public libraries at your disposal to choose from.
More than a Report or a Dashboard
A web app doesn’t have to be just a report or a dashboard, it can be a complete solution. There are technically no limits to what is possible. For example, an app can be a set of visualizations with two-way interactivity, other apps or reports/dashboard embedded in it, a place to allow users to run ML models etc. And if all of this could be developed & deployed comparatively faster and within one framework; sign me up!
Open Source Availability
These app development frameworks are based on Python or R which offer the full set of capabilities that these languages offer in terms of data analysis utilizing the open source libraries. With report or dashboard development tools, you are either limited to SQL and in UI features or in case of Power BI, to SQL, M (power query) and DAX. But with low-code frameworks, you can leverage any of the innovative and useful open source libraries. All algorithms, visualizations and data wrangling are enabled by open-source libraries that are available to the team in an integrated environment.
The Balance
The law of instrument’s cognitive bias applies here, and we have to be cognizant to objectively assess our solution options to ensure the right framework is leveraged. For us, this is another arrow in our quiver. The right solution for the use case would have to be used. For some use cases like a scorecard of metrics & KPIs for an executive, a PowerBI report is the perfect solution. For others like geospatial analysis of T-Mobile’s network coverage, we would have to rely on our software engineers to build us a solution using the full-stack frameworks. This framework of what I call ‘rapid app development’ using Python or R, gives our BI engineers and data scientists another technology to solve business problems.
We reviewed several options for this framework including Plotly, Panel and Bokeh (all open source) and even used Plotly Dash’s open source libraries for one our existing apps. The team then leveraged a 3-month long proof of concept to further evaluate these solutions. I will keep you all posted on how it goes with this exploration. If you have used these frameworks, let us know your experience in the comments.
