How to Reveal the Real Organisation Using Data You Already Have
Turning activity data from day-to-day work tools into interaction network topologies
Even though we’ve mapped and re-mapped every corner of our physical world, our organisations remain chronically opaque.
The way we tend to run them doesn’t help. But there is a deeper reason.
To “see” an organisation is to piece together multiple perspectives on a complex, changing entity, each telling its own story.
It’s a challenge we @ ConnectedCompany face frequently, usually with time in short supply.
It’s not unlike finding your way around a new city.
Street smarts, an open culture, good navigability and serendipitous discoveries all play a part.
But a good map is indispensable.
If you stick around, I’ll show you one of our go-to techniques for getting our bearings around organisations.
It makes use of activity data dispersed across our day-to-day work tools, the sort in widespread use across companies of all sizes.
We will use this data to create a bird’s eye perspective that shows individuals across the organisation interact as they go about their work.
When we’re done, the result will look like this:
This is a data-informed representation of the “people layer” of an organisation. One way to think of it is as the fabric for the flow of information, work and behaviours.
What we see reflects a particular period in time - though we are not limited to a snapshot. Comparing perspectives over time has much to tell about the evolution of collaboration and work, and the true impact of the changes we make.
Used thoughtfully, these networks are a uniquely powerful asset.
My hope is you’ll soon know enough to create your own.
In what follows, I’ve chosen to focus on the most important principles and nuances behind the way we create these views, rather than on a specific implementation.
Some data wrangling is inevitable, but I’ll aim to keep this general enough for you to use whatever tool or language you’re comfortable with.
You’ll also need a network visualisation tool or library to display the end result.
Let’s start with the data.
Gathering the data
As stated, our visualisations will be derived from readily accessible data stored within commonly found work tools.
In the past, we’ve used Slack, Teams, Jira, GitHub, CRM tools, Dropbox, calendars, email and others.
Each brings its own unique perspective on the world of work, so combining them often makes sense.
That said, usually we’ll start with a dataset from one tool. As long as it captures significant day-to-day work activity relevant to what’re looking to understand, it’s a good foundation to build on.
Each of these tools stores all sorts of data. But we only need two kinds:
- historical activity — a log of relevant things we did within each tool. This will form the basis for deducing the interactions for our network.
- additional information about the individuals and activities — we will use those to refine and make sense of the network topology.
Let’s illustrate this in the case of Slack. If you are not familiar, it’s a communication platform where users post messages and content in topic-based channels.
With Slack, our activity data is simply a list of contributions made to public channels by various individuals at different points in time. We can stick to messages only, or include reactions, file shares and other activities too.
That’s it. It’s just a feed of sorts.
The second kind of data will help us add meaning to our networks. The data is just a set of attributes for each type of entity in our activity dataset. It can come from Slack, or from elsewhere.
For example, for each individual, we may have things like:
… and for each channel:
That’s all the data we need.
(Incidentally, Slack makes it particularly easy to export this type of dataset. The is an Export Your Workspace Data feature, accessible via the admin console.)
The art of observing interactions
Our overall goal is to map the interactions between pairs of individuals. How does the data we now have help us?
Clearly, some activities within our tools directly represent interactions. A 4-person meeting in our calendar is an example, as is person A sharing a file with person B.
More often, though, interactions within our data aren’t as explicit. Say you and I post in the same channel on Slack. Is this an interaction?
It’s very easy to over-think this question. We might say that it depends on the topic of our messages. They may be about the same thing, or completely different things…. Or similar things, in which case we need to draw the line somewhere… And that line may need to be flexible, because what if our messages were posted days apart…
This is rabbit hole territory. What’s more, with every decision we make, every label we attach, every rule we set, we are potentially adding a layer of bias. We, the unfamiliar observer, wishing to leave no trace.
Luckily, we don’t have to go there.
We can do better by observing our own ability, as humans, to identify interactions visually, even when we can’t perceive their purpose.
Imagine walking the floor of an open-plan office. To the left, a group stands around a post-it covered wall, taking turns to speak. To the right, a meeting room, with five people huddled around a speakerphone. At the far end, a hot-desking area, a set of individuals each absorbed in their own work or call.
Now imagine returning to the same place at various times during the week.
You see the standup group working, sitting, talking together almost every time. You only see the five from the meeting room a few times, in smaller groups. On a couple of occasions, one of the standup people is with them. And in the hot-desking area, although the faces change frequently, interactions remain minimal.
At the end of the week, despite knowing almost nothing about the content of the communications between these people, sketching a simple network diagram is trivial.
For its creation, we used just two pieces of information:
- a shared interaction setting —we observed people in a common location, and used knowledge about the nature of each setting to identify groups
- individuals’ activity within that setting —we recognised that interactions necessitate active group participation
Next, we’ll take this idea with us from the physical world into digital.
Turning activity data into an interaction network
To do so, we’ll take advantage of another simple observation.
The way we organise work and information within our tools reflects the structures, initiatives and groups that exist within our organisations.
Our Slack channels usually reflect projects, teams, org units, communities, topics of interest. In Jira and similar tools, we usually organise our tickets by project or team. The events in our calendars reflect groups of people brought together through a common purpose or activity. Our CRMs have customers, deals, leads… Code repositories and document tools have folder hierarchies. And so on.
These constructs provide digital representations of the interaction settings we observed in the physical world.
And activities within those settings tell us about each individual’s strength of association with them — hence with others within them.
The standup team could frequently be seen in the area around the team wall. Their team channel on Slack would tell a similar story.
We will shortly use this principle of observing activities within shared contexts to extract and visualise interaction links between individuals in our organisation.
Before we do, let’s take a brief detour to reinforce the core concept.
Interlude: mapping the fabric for the spread of behaviours
In his brilliant work on the spread of behaviours and other “complex contagions”, Damon Centola used similar ideas to map and simulate emergent organisational networks.
Centola’s approach characterises each person in an organisation as belonging to a number of settings where people can meet one another, through which work and social ties form. The canteen, the project, the meeting room, the working group, the coffee machine.
Connecting people based on the similarity of these memberships forms the basis for an emergent network topology for the entire organisation.
Centola went on show how characteristics of these networks of social ties directly affect the diffusion of innovative practices and behaviours. He highlighted three network archetypes, with different characteristics.
In organisation A, all individuals have easy access to the entire network. In organisation C, on the other hand, localised links dominate, with little interaction across groups.
(Interestingly, it turns out that novel practices and behaviours, both of which require reinforcement, spread most readily through Network B.)
Centola’s approach to mapping network topologies will look familiar. It’s also a great illustration of the kinds of uses they can be put to.
Creating the network
Let’s return to the task of deriving interaction links from the activities within our interaction settings.
To extract these, we’ll need to make two more decisions:
- the specific activity settings to use to derive interactions, in cases where multiple options exist
- what significance (or weight) we should attach to each interaction link
To keep things simple, we’ll use rules of thumb that tend to work well out of the box. You are of course free to deviate, and encouraged to experiment.
Firstly, we need to choose the interaction settings that group together meaningful activities within each of our tools. This in part depends on how tools are being used — but is generally straightforward.
For Slack, for instance, channels are the natural choice — though we may drop #general, and any others that represent “global” contexts.
For GitHub, the repository is the natural choice — or top-level directories if you have a ‘monorepo’.
For Jira, usually a project, sometimes an epic. For calendar, a meeting. For CRM, a prospect, an account, or a deal… And so on.
Secondly, how do we turn activities within these settings into network links that connect pairs of individuals?
We’ll follow the intuition that there should be a strong link between any pair of individuals who are active in the same places on a regular basis.
A simple way to implement this is to calculate the average number of shared interaction settings per day for each pair of individuals. Count the number of times both contributed within the same setting on the same day, and divide by the number of days.
We can improve on this approach in several ways. Those will have to wait for another day. What we have here will do fine for now.
And so we finally get to a dataset of weighted node pairs, representing the links for the network:
Et voilà
The last step is to reach for a graph visualisation library or tool, and display the network. All of them take a list of (possibly weighted) links as input, plus whatever attributes for each node we wish to provide. Most can map these attributes to visual traits such as size and colour.
Here’s one rendered with the high-end Regraph toolkit, and its lovely layout engine:
Note how we’ve used attributes associated with the people in our network — in this case their employment status — to add meaning. Also note the clearly visible team clusters, as described in the caption.
For comparison, here’s another network, using the open-source nx_altair library:
Once we have the technique nailed, it is easy to take snapshots of the network for various time intervals, and visualise its changing shape:
But wait… all I got is this lousy hairball
Sooner or later, it will happen. Here’s a really hairy one. Yours probably isn’t as bad.
But don’t worry. There are two reasons hairballs happen:
1. We included too much information — the above hairball is based on activity data spanning three years, in a place with somewhat fluid and tightly coupled teams. This produces lots of weak links, which affect the layout adversely, as can be seen.
The first thing to consider is to reduce the timeframe. The second is to drop some of the less significant links.
With a narrower, more useful time interval, and a bit of pruning, our hairball begins to unravel, with several teams easily discernible.
2. We truly have a very dense network — Some time ago, we had high hopes for a dataset from a new client. All was going swimmingly, until the moment to unveil the network came.
We were left staring at an entangled mess, wondering what went wrong.
It turned out that nothing had gone wrong. The company’s core business involved running 3-month boot camp courses on site. They actively promoted teamwork, group projects, social interaction, networking among the participants. On most days, the entire floor was a hubbub of activity, with people moving from team to team, room to room. Everyone got to interact with almost everyone else regularly.
So “of course we got a dense network”, a version of Centola’s organisation A.
Context matters, and much of it lives outside the data.
Putting networks to work
Now that we know how to visualise network topologies, what can we do with them?
Time and again, we’ve found them to be highly impactful artefacts.
They embody the paradigm of a “connected company” that intuitively makes sense to people, a radical improvement on the lines, boxes and hierarchies traditionally used to represent organisations.
They carry with them the authenticity that comes from their roots in actual day-to-day work activity.
And they have an intriguing, often alluring quality to them that invites curiosity and further exploration.
Loosely, we’ve found people’s engagement with them to happen in two phases.
Initial impact is all about the representation, the orientation, the aesthetic.
It’s a new way of seeing the org. It’s often beautiful. People look for themselves in it, and for their surroundings.
And then they start to notice ways in which the network supports — or challenges— established perceptions. The “aha” and “of course” moments start.
This process isn’t analytical in nature. Its power is in the opportunities it creates for making better sense of how the organisation truly works, what it is, and how to think about it. These are seeds of transformative mindset shifts.
The second phase of extracting value from the networks involves deeper, more specific questions, such as:
Which nodes have important roles in the network?
How are our domain experts being used?
How interdependent are our teams, and how can we reduce this?
Why does it take so long to get things done?
Are our partners integrating well, and transferring knowledge?
How resilient are we to key people leaving?
What would a better team composition for us look like?
Are these things changing for the better or worse?
These kinds of questions call for a more focused, analytical approach. Naturally, the learning curve gets steeper, and forks into specialised areas as it does so.
The size of the prize grows too, with the power to better inform key decisions and interventions — org redesigns, strategy shifts, people and technology investments, business process changes and more.
We’ve already written about some of the steps we take to extract value from work toolchain datasets. You’ll find the strong network-centric theme highly relevant to the present discussion. We will write more as time allows.
For now, I’ll leave you with some of my favourite references that illustrate the power and allure of organisational networks.
- The Network Thinkers blog is a great place to start if you are new to exploring these networks. Valdis Krebs is a pioneer in social and organisational network analysis, and has assembled a great collection of accessible anecdotes on the blog.
- Team Topologies is a blueprint for organisational design, with interactions at its heart — interactions among teams, as well as between teams, software and domain knowledge. It embraces the dynamic, evolving nature of organisations, unlike most so-called scaled frameworks. We’ve had good results using activity data to explore inter-team interactions and various forms of coupling. More in a future post.
- We’ve touched on Damon Centola’s How Behavior Spreads already, but it’s well worth another reference. If you’re looking to better understand how behaviour change propagates, and ways to approach interventions, it’s a must read. If you only have an hour, I’d recommend this talk.
Final thoughts
Mapping and interpretation of organisational networks belong in thoughtful and responsible hands.
Each network is a series of statements about real individuals, their work, their teams, the organisation itself. Without due care, these statements can be misunderstood and misused.
Context is crucial, always. A person acting as a bridge between two groups could be holding key relationships, buffering unwanted interference, co-ordinating projects, exerting excessive control, or something else we haven’t even considered. From the network alone, we just can’t tell. Examples like this abound.
Reflecting this, at ConnectedCompany our use of network topologies never takes place in isolation. It proceeds alongside active organisational context immersion. We talk to and work together with people across roles and levels. We get to know the nuances of how their work tools are used. We pick up further context from thorough exploratory analyses of the datasets we extract. We don’t leap to conclusions.
If this is your first foray into visualising network topologies, your own organisation is the place to start. You and those around you are already familiar with it. This is a great head start as you begin to make sense of things.
It is also crucial to accept and embrace the inevitable imperfection of network topology representations. Any representation of an organisation is destined to be incomplete. Organisations are complex, messy, multi-faceted entities, where the very notion of a complete representation defies definition.
Imperfect as they might be, our models and representations can be immensely useful. We just need to treat them for what they are, investing deliberate effort into understanding their foundations, their nature and their limitations.
After all, we are all reliant on our own imperfect models of reality. We always have been. We started assembling our own mental model of each place we worked at the moment we arrived. We interacted with others, examined org charts, looked around as we walked the floor, read announcements, picked up on the gossip…
Against that backdrop, perspectives based on work activity data are a powerful yet massively under-explored addition.
So go ahead and make a start. Get some data, plot some networks, see what they show. With a curious and thoughtful mind.
You’ll have fun, and, we hope, make a company better.
