Introducing Paradime
Introduction
Today we are lifting the curtains and introducing Paradime, the operating system for analytics, that we have been building for the past 2 years in stealth.
Building a startup in stealth is hard. It’s harder than you can imagine. Hiring is hard, prospecting is hard. Talking to people about your product is hard. But it does help build and iterate a product without distractions alongside our design partners to firm up the proposition. So that’s what we’ve been up to.
In the modern data stack, over the last few years, the proliferation of new point solutions has led to tool chaos. There are so many tools out there that it’s impossible to keep track of them all. Data leaders are spending more time than they should trying to manage their tools, costs and vendors. They are tired of making round-trips with procurement. They’re also sick of dealing with data sprawl across multiple vendors, which increases the risk of data breaches.
For the analytics engineer, the tool chaos has led to cognitive overload and a drop in productivity. Life has become harder than ever before.
As Benn summarized in his post The powder keg of the modern data stack that:
The biggest looming battle, however, will be over a different territory: The brain — or operating system
At Paradime, we are building the operating system for analytics 🥳 .
Background
Back in 2018–19, at Octopus, my team was rebuilding the entire data stack from scratch. We ripped apart a brittle legacy stack consisting of SSIS, bespoke ingestion code written in .NET, SQL Server, Qlik etc.
The final solution that we ended up with after 6 months of gruelling work looked something like below:
- Stitch, Fivetran and Segment for data ingestion
- dbt-core running inside airflow for transformation
- Looker for visualization
The community had not yet coined the term Modern Data Stack. It used to be called the ELT framework 😀.
On top of the tools architecture, when we added the people/team layer, we saw that:
- Data flows kept the tools tightly coupled
- Yet the tools fragmented the people / team layer
- And, Slack was the only loosely connecting glue for human communication
Since 2019, the explosion of the Modern Data Stack has also exploded the tools that work on top of the warehouse. As these tools came into being, the stack became even more fragmented. The people and their attention connected to the tools also became fragmented and unbundled.
Every data analyst would spend most of their days bouncing between low-code/some-code/open-source / commercial apps, fighting fires and drowning in data requests when all they should be doing is generating RoI for the business.
An average Series C+ organization would have 70 employees in business functions per data analyst, which means on any given day these 70 employees would be going nuts that they don’t get answers to their questions or requests and the poor analyst would be screaming in his head that he wanted to build valuable insights not respond to Slack all day long. This deadlock we saw first hand at Octopus, then at the Guardian, then at Revolut, Hubspot, Carta, and the list literally goes on.
We heard time and again from data analysts and analytics engineers that while dbt had liberated them from SQL hell, the explosion of tools have driven them back into hell-fire.
On the other side, we saw that business functions hunger for data and making data-driven business decisions that increased exponentially. Slack without data context was no longer fit for purpose for the data-driven enterprise.
The essential human to human conversation layer around data was clunky, time consuming and devoid of context.
We decided to tackle the problem of productivity and collaboration in analytics workflows through Paradime.
We realized that the new world of dbt + the modern data stack needs a new category of tools to work with so people can work faster, smarter, and a lot less stressed.
What are we building?
The data analytics discipline has evolved massively in the last few years. There is a movement to bring software engineering principles to analytics. Analytics as a discipline is different from software engineering. There is code context, data context, and people context. Yet, as analysts, we are stuck with tools used by software engineers.
We are changing this status quo by bringing to the market the following:
- An analytics operating system that gives people control over their data while creating a non-linear value jump through efficient workflows.
- connecting data, analytics, and business functions in a single collaborative workflow
If we take the blueprint for modern business intelligence from @A16Z below, each of the boxes in the diagram represent a category of tools of the Modern Data Stack. Each tool represents a source of compute, where something happens with the data. But then there are arrows connecting these boxes. These arrows represent people, processes, and productivity sinks.
It’s like having really powerful processors on the motherboard while the bus system between them is limited in throughput.
Paradime is built to super-charge those analytics workflows, which today are either non-existent in most orgs or only present in businesses with significant resources to build internal tooling.
To that effect, today we are announcing 5 components of that workflow as explained below.
1. Onboarding
Shifting away from setting up and managing dbt workspaces on individual laptops, in Paradime you can onboard analysts in less than 3 mins once the account is set up by the admin. Admin account setup takes less than 30mins and does not require any engineering support. There is no 3 month implementation and professional services costs.
We support connecting to dbt repository on Github, BitBucket, and Gitlab. We support connecting to Redshift, BigQuery, Snowflake and Firebolt and more.
2. Code IDE
The Code IDE is the crown jewel of the Paradime experience.
The Paradime IDE brings best-in-class desktop IDE experience for analytics to the cloud. It’s fast, performant and has the widest coverage of features. It’s purpose built for analytics workflows compared to general purpose cloud IDEs such as Gitpod, AWS Cloud9, Stackblitz, which are more suited for software engineering.
It comes with all the ergonomics that developers expect from a desktop IDE, but some of the notable features include:
- Everything you have in VSCode — look and feel, keyboard shortcuts, file search so you don’t have to learn a new IDE from scratch.
- A native terminal to run any CLI command including git, python, and SqlFluff
- dbt power users can install their own python packages too.
- Full UI-based git-ops support for beginner and advanced users
- And most importantly, real-time in-line view of lineage, docs, data-preview for any dbt model — without running any dbt command or generating manifest.json
If you ever felt stuck with a dbt-cloud IDE or intimidated by the complexity of local setup, now you have a choice. We are taking the IDE experience to a whole new level with plenty of ground breaking features coming next year.
3. Graph Lineage
The problem we wanted to solve here was:
- Analysts cannot wait hours or even minutes to see an updated lineage as they build new models or refactor existing ones. They need a real-time view.
- Analysts need a holistic understanding of the lineage in both the dbt layer (already in dbt-cloud) and the BI layer in their day-to-day work.
So we built the Paradime Graph lineage that updates with every commit in your dbt-repo in real-time. It spans from your data sources all the way to your dashboards and reports. We support both Looker and Tableau with more coming on the way.
For Looker, we provide lineage across Views, Explores, Looks, Dashboards, and Schedules, providing you with an end-to-end view of your dbt+Looker lineage. Similarly, for Tableau we can link Data Sources, Worksheets, and Dashboards.
Teams are moving beyond dbt exposures , which are hard to maintain and does not provide any visibility on the components of the BI layer between dbt tables and BI dashboards.
4. Bolt Schedules
The problem we wanted to solve here was:
- Analytics users find Airflow / Dagster / Prefect hard
- Data Platform teams don’t like people going and changing production workflows in the dbt-cloud UI with no traceability.
In Paradime, you can author workflows using a simple YAML format that is git-tacked. You can set up notifications across Slack and email. Finally, you can view results from production runs in the UI. We also have an integration with AWS S3 so we can pipe all your data back to your own S3 bucket and you stay in control of your data.
We also have APIs to trigger your dbt schedules from Airflow, Dagster or Prefect and receive alerts when they are complete. The API gives platform teams more control to manage dependencies upstream and downstream of dbt schedules.
And did I tell you — we also have a one-click importer for all your dbt jobs from dbt cloud to Paradime, so migrating does not feel like a crazy adventure.
5. Query SQL
Building a dbt model is most times the last step in the modeling process. Analysts spend significant time exploring data in the warehouse and fiddling with raw and compiled SQL of existing models to run their daily work. There are numerous back and forth between the IDE and SQL editor, copying and pasting, editing and replacing table names with refs.
The problem we wanted to solve here was:
- How to make the dbt modeling and data exploration process seamless so you don’t have to go back and forth.
- How to make tuning compiled dbt models efficient without having to make roundtrips between your code editor and SQL editor
The focus here being on flow and productivity.
Who is Paradime for?
During the evolution of OS for Apple, there were two schools of thought — Wozniak believed that it should be an open system so hobbyists and tinkerers could play with it and Steve Jobs, thought it should be a system that just works. Today, we love MacOS, because it just works.
During our research, we found that there are similarly two main personas of analysts / analytics engineers in the world today:
- Tinkerers — These are people who love to tinker, play around with different tools, and libraries. They like to explore and build their own features on top of existing OSS. They are already super comfortable with their own IDE setup. They are hobbyists. They are more software engineering orientated. Paradime is unlikely to be a great fit for them today.
- Builders — These are users who are singularly focused on getting their work done quickly with zero-hassle. Users who value productivity, users for whom velocity wins. They want to hone their analytics and data modeling skills. They want to generate business value from their insights. They just love tools that need zero setup, that just work. Paradime is for them.
We are in a public beta phase, so you can’t sign up straightaway just yet on Paradime and there are a few rough edges, but if you DM me on LinkedIn, comment here, I will sort out a 30-day trial for you without any credit card of course.
What’s next?
Well, a hell of a lot. We are announcing just the tip of the iceberg here. We are also launching on Product Hunt next week, so please follow us to get notified when we launch.
We have a lot to come in the coming months, so hold on to your seats. 🚀
Because as I said before, the world of analytics really needs a better class of tools to work with and we have a long way to go.
