Applying doctrine to data flow

Up until now, we have been stepping through Simon Wardley’s brilliant work on Wardley Mapping (https://medium.com/wardleymaps). We’ve looked at a basic value chain for “real time business automation”, applied that value chain to a Wardley map to see both position and movement, and we’ve looked at some basic climatic conditions that will influence future behavior.

The next step in Simon’s basic process loop is to determine (or reaffirm) basic doctrine with respect to how we “play the game” from this point further.

Here is how Simon describes “doctrine” in this context:

Doctrine are the basic universal principles that are applicable to all industries regardless of the landscape and its context. This doesn’t mean that the doctrine is right but instead that it appears to be consistently useful for the time being. There will always exist better doctrine in the future. As with climatic patterns we will go through some basic forms and refine in future passes through the strategy cycle.

The important things to take away from this definition are:

1. Doctrine isn’t set in stone, but is none-the-less “consistently useful for the time being”. This means doctrine doesn’t change with every single shift in the wind, but is flexible enough to adapt as core assumptions are challenged.
2. Thus, future doctrine will adapt. It has to. New technologies, new markets, even new political realities will make fundamental change inevitable.
3. One key word in that definition, is “refine”. Tailor doctrine to changing needs, but — and this is important — understand that you aren’t tailoring to your own sales and marketing needs. You are adapting to external realities…e.g. “disruptions” and new user needs.

Now, I’m going to do something a little weird as I step through Simon’s doctrines (which may not necessarily be exactly your own company’s doctrines). I want to point out the weaknesses in the assumptions I made, and the generality of the use case I used, as well as the strengths. I do this to be clear that you can’t just take my map and say “there it is, the universal map for what we should do next”. In fact, you should define your own doctrines and find the ways in which my maps (and analysis) fails you.

That’s part of playing the game.

So, let’s get started. (For definitions of each doctrine, see Simon’s post.)

Doctrine: Focus on User Needs

This doctrine is a perfect example of why the map I’ve created in the previous posts is both useful and dangerous for making decisions about how to address this nascent market.

I stand by my assertion that the reason for enterprise software is, in fact, to automate our economy, and that real-time is the next logical evolutionary step for this purpose. However, it’s a horrible “user need” to use to find any specific value opportunities. It’s much too general. It’s not even really a “user need”, but more a category of user needs. It’s not what anyone means when they talk about identifying a specific user need.

So, how do we go about finding a real user need to map? One common technique used in a variety of design and analysis use cases is the “5 whys” approach, in which you start with a stated problem and ask a sequence of questions to get to the root need underlying the problem. So, find an actual user, ask them about their perceived pain or wants related to a problem space, then start breaking that down further by asking “why”.

(Finding actual users can be tricky, however. For example, often the buyer of a product or service is not the same as the end user — such as when IT operations buys a management platform for use by developers. If you ask these questions of the wrong “user”, you’ll get the wrong core needs and build the wrong solution.)

Another approach is suggested by Simon in his post on doctrine: start by looking at the transactions an organization makes with its customers. This can even be combined with a technique like the 5 whys to get at hard to articulate unmet needs.

A better user need here might have come from analyzing the core needs for a specific data stream, or a specific business entity that requires real-time interaction with their digital processes.

Doctrine: Use a common language

This is a doctrine that’s reasonably easy to meet in this series. I am creating maps that I am using to step you, the reader, through my thought process. The map, and the language I use in my posts, becomes a common language for this purpose. Of course, as Simon points out, if you use these techniques in a more traditional manner, you may want to work across actors to define a single consensus map.

The one area of “language” that I’ve struggled with, however, is what to call the general category of applications we are talking about here. I’ve started with the term “data flow”, which in my mind means more than any specific architecture, but includes the cultural, organizational and practice changes that will be driven by a stream or event centric approach.

The preferred terms as of this writing seem to be “event processing” or “stream processing”, but I don’t see a grand consensus as yet. However, if a term starts getting seeing significant usage across the technology and business community, I will choose that term over “data flow”. Using common language is extremely important for describing emerging trends, but also generally difficult and contentious, and this topic is no exception.

Doctrine: Challenge assumptions

This one is on you, the reader. I write to learn. I want to be told I’m off my rocker. I want you to challenge my assumptions, my analysis and my conclusions. Doing so will ensure that both of our models of data flow stand up to future scrutiny (or start from a good place if they need to change).

Doctrine: Remove duplication and bias

Generally, if you are evaluating technical strategy across a large organization, and drive out biases about skills, technologies or climate, you will find significant opportunities to remove duplication of common needs. In this case, however, I’ve focused on an need chain that narrows down the value chain in a way that removes much potential duplication already.

That said, there is a lot of duplication in the underlying technologies already. OSS projects like Kafka, NATs and OpenWhisk compete with similar services from Amazon, Microsoft and Google — not to mention “custom” options developed in house. Data streams with common information, such as weather or personal health measurements, can come from a number of different sources.

Thus, duplication in your own organization’s execution of data flow is a strong possibility. Often, as the first projects are explored (usually a “genesis” activity), it’s fine to see some capabilities duplicated. But, as architecture adoption increases, good strategy dictates that application teams focus on doing as little duplication of existing capabilities as possible. The map (and some of the other techniques that Simon outlines) can help assure this is the case.

Doctrine: Use appropriate methods

One of the key things our maps can demonstrate is where you can use various methods for implementation and operation of the overall value chain technologies. Simon’s post uses the following legend to explore how a simple value chain can be divided between “outsourcing”, use of off-the-shelf products, and custom development.

So, let’s apply something similar to one of our core value chains:

Now, I fudged a bit here, and assumed the lowest option based on evolutionary stage for the two items that are on a border (“Streaming Protocols” and “Container/Server Platform(s)”). Either of those could easily be done with the next most “standardized” technique. However, as you can see, the argument Simon makes about “no one-size fits all” for development and operations methods holds solidly for our “streaming integration” user need.

Doctrine: Think small

Key for our analysis is the concept of not thinking of a data flow architecture as a single deliverable, but a system made up of many composable pieces. The more composable the system components, the more flexible the architecture for future use cases.

Think of the Linux command line text processing tools. Each application is pretty damn tiny (even sed and awk), but really sophisticated scripts are written with them every single day, for a wide variety of use cases. The key to this success is each tool is focused on a specific responsibility, and integration is assumed to be possible based on a standard integration protocol (ASCII, generally).

I think one thing that our maps show clearly is that data flow automation requires some base components that do their specific tasks very well (data capture and data distribution, for instance) but has a huge blind spot when it comes to many of the processing algorithms and tools required to make those data streams useful.

If you try to build the “one great platform” for data flow, you will perhaps have some success early with shops that are scared to death of the continuous work and thinking required for working composed systems. But history tells us the long term belongs to a variety of standardized, focused building components. So, take that into account and build your systems for “composition first”, using techniques like micro-services, and standard functions and practices.

Doctrine: Patterns can be applied across contexts

I include this doctrine for the simple reason that this is exactly what our choice of “user need” and subsequent analysis is meant to demonstrate. While your specific user need might result in some additional value chain components, or different placement of the components we did cover, I’m reasonably sure that your own analysis will largely end up looking very similar.

This is the secret of finding common “platform” components in computing. Find the things that are common in a variety of contexts, and build for those most common elements. This is why, in my opinion, all three major cloud providers already have gateway (data capture), event streaming, and functional platform elements.

Other doctrines

For some ideas on other doctrines you can consider as you look at our maps, check out Simon’s post. For example, he includes the following incredibly useful table:

I’d love to hear from you which doctrines you think deserve more scrutiny. I’d also love to hear from anyone that thinks I’m way off base here. As I’ve already noted, I write to learn. If you have anything to teach me, comment below, or find me on Twitter at @jamesurquhart.