Finding the right socio-technical balance
Socio-Technical Balance in Data Mesh Implementations
Disclaimers:
- These observations and suggestions are personal observations and do not represent my current or previous employers. Any resemblance is purely coincidental.
During various conversations and panel discussions, I have been asked many times whether the implementation of data mesh is an organizational change or (yet another) technology-only initiative. I thought to capture my experience in this article from the lens of socio-technical architecture (as I have understood and perceived it).
A few years ago, I was introduced to the word “socio-technical” and my first reaction was an allergic one. Since then, with the experience gained from implementing global projects and organization designs as well as learning from industry thought leaders, I have come to understand the importance of proper balance of socio-technical elements. Before I share my perspective on data mesh implementation, here is a quick recap of what this socio-technical system & architecture is and how I have used this in this article
Socio-Technical Definitions
Here are a few (can I say widely accepted?) definitions of the socio-technical systems and socio-technical systems:
“Sociotechnical systems (STS) in organizational development is an approach to complex organizational work design that recognizes the interaction between people and technology in workplaces. The term also refers to coherent systems of human relations, technical objects, and cybernetic processes that inherit large, complex infrastructures. Social society, and its constituent substructures, qualify as complex sociotechnical systems.” (Sociotechnical system — Wikipedia)
“Sociotechnical Architecture is about taking a holistic co-design approach to technical and organizational systems, given the inherent impact they have on each other.” (Introduction to Sociotechnical Architecture: Why & What is it)
There is plenty of content on this topic, you can find some of the links below in the reference section.
Why is it important?
From the research and practical implementation experience, one can see that:
- People and technology are mutually dependent on each other.
- We cannot successfully deploy digital solutions looking only at the business side of things and involving only business people and vice versa, focusing entirely on technological
- The disconnect between those two results in reduced efficiency of technological processes and less business value delivered in a given time frame at an increased cost.
- Moreover, the disconnect also results in reduced motivation for everyone involved, which also leads to reduced productivity and less business value delivered.
- Therefore one should not be limited just to technology landscape management or technology aspects of such initiatives.
- There needs to be a proper balance between the social system and the technical system. The optimal performance is reached when the two are in the right balance.
- There’s no easy, prescriptive way to define this balance and it has to be found iteratively by the joint team of technical experts and business experts.
How do I look a the socio-technical stuff?
As I stated in my previous articles, in my humble opinion, enterprises can reap the value from data by digging deep into the people-process-technology triangle:
To keep things simple, I have captured various activities associated with the people, process, and technology and classified them into social or technology categories to quantify my findings, to determine the right balance in the implementation of data mesh. For instance, organization changes, people and skills-related activities are categorized as social, while any implementation of technology, or automation is categorized as technology.
PS: I must admit, the process part is tricky. Is it social or technological? For the sake of this article, the definition and alignment of the process is classified in the social category while implementing it or automating it is classified as technical.
What is the proper social-technical balance for implementing data mesh?
One of the things that differentiate data mesh from various other “waves” of data warehouses, data lakes, lake houses, fabrics (and more) is a well-rounded approach by design that Data Mesh introduces in the shape of the 4 principles. Hence data mesh has been described as the “socio-technical paradigm”.
Below, I have attempted to quantify this balance based on my implementation experience as well as listening, learning, and collaborating with other practitioners, who have tried or have implemented data mesh:
Let me double-click on each of these principles in the form of a non-exhaustive list of activities within each principle:
Domain Oriented Decentralization
Socio
- Engaging the business and IT leaders to define the domains and their boundaries
- Identifying the business outcomes, vision, and challenges at a high level
- Evangelizing the data mesh concept and getting the buying from business and IT
- Identifying the domains within an enterprise
- Defining the domains, their domain context, and boundaries
- Defining the data domain leaders’ accountability
- Identifying the data leader for the domain
- Identifying sub-domains and possible product teams
- An initial map of data entities within the domain
- Identifying the various personas in the domain
Technical
- Implementing a “domain” structure for the infrastructure
- Setting up domain-related glossaries, definitions, scope, etc. on the data cataloging solutions.
Data as a Product
Socio
- Defining detailed business outcomes, challenges, and end-user personas
- Identifying the hypotheses and possible solutions and use cases
- Expanding and crisply defining and enriching the definitions of the data entities
- Defining the data product team structure (if new)
- Define & evolve the product roadmap
- Defining the UX design of insights (if in scope)
- Capture the value proposition of the data products
- Capture the SLAs, SLOs and SLIs
- Defining the data architecture and modeling and its evolution
- Investing in the teams’ development to understand product management
Technical
- Implementing (and/or reusing) the data architecture and data model
- Implementing the appropriate data architecture. ETL/ELT, curation, etc.
- Data engineering activities
- Data science activities
- Data visualization activities and front-end UX development
- Development, Testing, and Deploying (the DataOps)
- Product Operations and life-cycle management
- Implementation of governance policies (Access, security etc)
Self-Service Data & Analytics Infrastructure as Platform
This might come as a surprise, but organizational and mindset changes required by the platform team to treat the platform as a product are critical for data mesh implementation, and hence even within this principle, the social aspects are of critical importance and platforms is not just about technology choices & implementations.
Socio
- Creating a product team that manages the data & analytics infrastructure as a platform
- Establishing the product management mindset for the platform team, including investing in the teams’ development to understand product management
- Learning the DevSecOps and Platform engineering practices
- Establishing enabling teams to onboard the data product teams on the platform
- Establish the community of practices for the platform users (data product teams & other personas)
- Defining Show-back or charge-back mechanisms & processes
Technical
- Defining the thinnest viable platform and establishing a product roadmap for the platform in the shape of services->capabilities-> technologies
- Creating the development experience and automating the processes to reduce the cognitive load for the data product teams
- Implementing the domain and product structure in the underlying technologies to create the development experience environments
- Making sure that the technology choices seamlessly work with each other without manual work and integrations
- Implement FinOps and provide cost transparency to the data product teams
- Ensure that the platform enables and complies with the various internal controls and industry regulations and standards
Federated Computational Governance
Socio
- Establishing the federated governance with the inclusion of all domain leads and creating a forum, which is not the traditional bureaucratic governance bodies
- Driving the mindset shift to focus on business outcomes and value creation (even defining what value creation means)
- Defining what does “hub & spoke” model means and what value hub creates and what spokes bring to the table.
- Considering the environment & industry the enterprise is operating in, various external laws and internal compliance policies will define the various governance aspects of data. Developing an enterprise-wide understanding of such regulatory and compliance needs and translating them into appropriate data governance policies and procedures
- Creating domain and cross-domain forums to share the data stories, successes, and lessons learned. Encouraging and role modeling data sharing and re-use.
- Creating and role modeling the culture of data sharing and focusing on business value creation
Technical
- Translating the defined policies and procedures into the technical implementation at the platform level, so that every data product team can implement the policies in a consistent manner without the need to reinvent the wheel
- Providing cross-domain visibility on the work happening across the enterprise with the data mesh experience plane that allows visibility on SLAs, SLOs, and SLIs for platform and data products using fitness functions
The above are just a few examples and by no means an exhaustive list of activities needed. As I continue to learn every day by practicing and learning from fellow practitioners and thought leaders, I am curious to see the state of socio-technical balance in 6–12 months.
I hope these points will resonate well with others who are implementing data mesh and data products, please feel free to chime in and share your experiences as well. What balance have you found? For those, who are thinking of stepping into this area, I hope you find these points useful as you prepare to make the journey.
References:
- Sociotechnical Architecture — people and resources
- Primary Sociotechnical Design Heuristics
- Open Agile Architecture
- Marco Consolaro — Soulful SocioTechnical Architecture (video)
- Soulful Socio-Technical Architecture
- Sociotechnical Architecture: co-designing technical & organizational architecture to maximize impact
