Persona-Driven Data Architecture

From consumption patterns to customer obsession

Data engineering platforms can be likened to refineries. In the lifecycle of creating a consumer good, the refinery is where raw materials are processed and converted into usable substances, eventually creating consumer goods. The visual that comes to mind is industrial, with piping and instrumentation, and the refinery is typically not associated directly with the end customer.

Like refineries, data engineering platforms are designed to handle high-volume and high-complexity tasks, behind the back end. End-user experience is typically not a major discussion point associated with the design and maintenance of refineries, or modern data platforms. The structure of the system prioritizes optimal product output. Sure, eventually the product makes its way to end consumers, but that’s so far downstream that it’s not a concern for the refinery designer. The refinery designer receives requirements in the form of throughput, total processing volume, and sub-processes.

The Information Age is all about data. “Data is the new oil” contains a lot of truth and reinforces the refinery analogy. Refinery purpose and design align closely with modern data platform purpose and design. At least, in most cases, this alignment holds up.

However, in a time of generative AI, big data access has never been more widespread (and it continues to expand). The result is a wider variety of users leveraging data in a more diverse set of ways at more points in the data refinement process. This means that the end user is brought closer to the modern data platform, and all of a sudden there is a divergence from the refinery analogy.

The Evolution of Big Data Initiatives

Around five years ago, most modern data platform initiatives were focused on getting data out of legacy systems, to avoid license fees that came with contract renewals. The push to modern technology was all about shifting capital expenses (CapEx) to operating expenses (OpEx), and the end user was unaccounted for in decision-making. The only real “user” considered at this time were the operations and maintenance teams responsible for keeping these systems running smoothly.

From there, the shift was toward making better use of all that data being stored in that shiny new technology. Data scientists and business analysts entered the picture as users of the modern data platform whose needs were considered when designing pipelines and processes. The majority of consumption use cases were still internal, be it for developing big data models or executive reporting data visualizations.

Over the past year or so, I’ve observed an uptick in the trend of enterprises going beyond internal users to serve a true end user: the customer. This trend is correlated to the strong push to embed GenAI in client-facing products, but it’s not limited to virtual assistants. Organizations across industries are embedding data meaningfully into their products, and it’s not just the early adopters.

Process or Product?

Data engineering is largely more process-focused than product-focused, especially when compared to more feature-oriented forms of software engineering, such as application development. This is why the refinery analogy holds up so well (and again, why “data is the new oil” is such an accurate saying). However, when the customer is the consumer, there is a mindset shift from process to product. Yes, the data engineering work involved in getting the data into the product is still a process-heavy job, but all of a sudden the presentation, responsiveness, and customization available to the end user become key topic points.

Slalom’s experience design (XD) and data engineering (DE) teams do not often have the chance to collaborate closely on client engagements. Recently, one of those rare opportunities occurred when working with a software product company.

Throughout the requirements gathering process, it became clear that the product development our XD architect was driving and the modern data platform design our DE team was owning were not separated by very much, if any, software engineering (SE) back-end or front-end work. Recognizing we were both in unfamiliar territory, we began collaborating closely.

Put Yourself in Their Shoes

Early on in our discovery, our XD architect did something seemingly basic yet profound (at least to my DE brain). While we were asking about product database insert, update, and delete processes (among other related topics), they went ahead and created a trial account on the product. It sounds basic, but it was undoubtedly the best way for us to understand how changes propagated through the system (and were returned back to the end user).

Our methodology involved gathering data platform requirements via wireframes and persona journeys. As our client identified the components they wanted to see in their product, we identified the data required to serve the desired front end. This involved working our way backward from the user interface and forward from the available source data to define data pipelines and transformations. “Left to right” (source to user) and “right to left” (user to source) were commonly used terms in our team vocabulary.

Not only did we identify the datasets and transformations required to deliver the functionality, but we also defined the data processing and modelling based on the desired user experience. For example, if we needed a certain component to load in under a second, we knew that value would need to be pre-calculated. If we wanted to enable flexibility and customization for the user, we had to make the minimum amount of that dataset available so the user could do what they wanted in a performant manner. With the flexible pattern, we also discussed caching the user-created data configuration for improved responsiveness after the first instance.

A lot of this was driven by the subject matter expertise of our XD architect. We were mainly interfacing with product managers, a common stakeholder for XD but typically less so for DE. Our XD architect could speak the same language as the product managers in order to get to the heart of the requirements for this engagement; this work was critical to our team success because it enabled us to define what success actually meant.

Another impressive aspect of working with XD was the way they approached discovery. They went beyond conversational questions and answers and used visual aids to drive (and record) the discussion. Fundamentally, this meant that discussions could spin off into tangents, which is often beneficial for requirements gathering, while providing a way to efficiently bring us back on track. Icons were used as visual cues to indicate the type of discussion point (for example, an apple with a worm in it to denote pain points or a crystal ball to mark future-looking), which added visual interest but also made it easy to go back to these notes and quickly find what we were looking for.

Our team entered the discovery phase with an idea of what success looked like. Our team “left” the discovery phase (note: discovery never really ends) with a much different and more accurate idea of the definition of success. This necessary evolution was driven by effective requirements gathering. In turn, we were able to stay in sync with our client and successfully delivered a platform that benefitted not only our client, but also their customers.

The Collab We Never Knew We Needed

While XD-DE collaborations are rare, I hope they become more common as part of the next wave of big data projects. Personally, I have never had such a thorough understanding of the data lifecycle; the difference here was the intense focus on the customer. After this experience, I have become a champion for XD presence on every DE project, at least for the requirements gathering phase.

As technical siloes break down with increasingly interconnected and accessible tools and systems, our people and processes need to adapt accordingly. Ultimately, even the most back-end data refinery is intended to serve a human. Deep understanding of the end user experience is key to delivering the best data platform. Our customers are more than consumption patterns.