Omicron Paradigm: Architectural patterns for the Infinite Data Logistic
An Introduction To Omicron Paradigm
Many enterprises are increasingly relying on vertical & horizontal SAAS applications to operate their business. The modern enterprise depends on SAAS applications for all business operation touchpoints from customer relationship management, marketing & demand generations, human resource management, finance and accounting, content management, support ticket management, supply chain management, inventory management, and digital & multi-channel selling.
The specialized SAAS applications provide significant vertical business insights to a business to operate efficiently. Salesforce strategically has a better understanding of the CRM domain, where Zendesk has a better understanding of the customer feedback domain. The contextual data enrichment, such as sending the support ticket information from Zendesk to Salesforce or attach the product data to the support tickets, can significantly magnify the SAAS applications’ ability to provide much deeper business insights.
Seamless integration among the specialized SAAS applications can exponentially increase the operational excellence of a business. Hence, there is a hidden infinite logistic of the data from one SAAS application to another. I call it the Omicron Paradigm.
An enterprise’s success depends on how efficiently it manages data flow from one SAAS product to another. So, in a way, Every company is a data-logistic company.
Omicron Architecture Patterns
Some exciting patterns are emerging from different corners of the industry to streamline the Omicron paradigm. We can broadly classify the Omicron architectural patterns into three patterns.
- Contributor pattern
- Proxy pattern
- Orchestrator pattern
Contributor pattern
The contributor pattern is the most widely adopted Omicron pattern. The pattern is the continuation of the traditional ETL model, where the data ingested from different data sources to one centralized data lake and flowing to various other destinations. Thus, you can conceptually imagine the SAAS applications as an extension of the traditional data mart.
The contributor pattern’s advantage is that the data ingestion tools like Fivetran, Rudderstack, and Airbyte integrate with most SAAS applications. In addition, the reverse ingestion from the data lake to the SAAS applications is gaining traction with Census and HighTouch.
The commoditization of storage by the cloud infrastructure accelerated the contributor pattern; simultaneously, it brings significant data management challenges. The data ingestion frameworks can abstract the infrastructure to ingest the data. Still, the data management, quality, correctness, and integrity of the data fall into the internal data platform team. The contributor pattern’s success depends on highly skilled data practitioners, a challenging skill set to acquire for an enterprise.
Proxy pattern
The proxy pattern is the logical next step for the cloud data platform’s evolution by providing a data management layer abstraction on top of the cloud data warehouse which acts as a proxy to manage the data logistic across the SAAS applications.
The critical difference between the proxy pattern and the contributor pattern is the cloud data platform manages the data ingestion and simplifies the data management practices. As a result, the enterprise data warehouse simply another SAAS application without building a complicated data management platform and SAAS integrations. Snowflake’s data-sharing platform and Databricks data-sharing platform are some of the trends following the proxy pattern.
The proxy pattern reduces the data management complication and enables an enterprise to set their business operations fast & efficiently. Data is a critical competitive asset of any enterprise. How the cloud data platform can provide data portability and platform portability are challenges while adopting the proxy pattern.
Orchestrator pattern
The orchestrator pattern encapsulates the recent trend from the SAAS products to provide the Data As A Service (DaaS). The recent acquisitions from Atlassian(ChartIO), Google (Looker)& Salesforce(Tableau) amplifies the SAAS DaaS offering. Simultaneously, the advancement in federated query engines like Presto/ Trino, the domain ownership & the data mesh principle, and the No-ETL vital contributors to the orchestrator pattern.
In the orchestrator pattern, A cloud-native distributed data platform enables federated data access across different SAAS applications. For instance, some work in the Trino community integrates the Salesforce JDBC connector with Trino.
The Orchestrator pattern significantly simplifies the data management by pushing the complexity to the SAAS applications. As a result, the SAAS applications have better domain understanding and can solve the data management problem for once than each enterprise solving independently. However, the lack of standard protocol to access the SAAS platforms and the federated query engine’s efficiency in joining multiple data sources are challenges in adopting the orchestrator pattern.
Conclusion
The Omicron paradigm is an exciting development in enterprise data management. There is no one silver bullet architectural style for success here, as each architecture has its maturity model. Data integration & data management remains the challenge for any modern enterprise. Commoditization of data management by embracing any one of the Omicron Architecture will shape the data technologies in the future.
All the logos are the property of their respective owners and are used here for discussion purposes only. Links are provided for informational purposes and do not imply endorsement. All views expressed in this newsletter are my own and do not represent current, former, or future employers’ opinions.
If you want to stay updated with all things happening in data engineering, please join my dataengineeringweekly.com newsletter.
