Development and Deployment: The Human Perspective
tl;dr Developing and deploying IoT solutions in the enterprise is challenging. Organisational structures are often not compatible with cross-functional topics, plus the interdisciplinary nature of IoT calls for engineers with a broad skill set and vocabulary. This post doesn’t offer solutions. ;;;
thingmonk 2017
In February the organisers of the thingmonk IoT conference asked me if I wanted to come back for another talk. As a regular speaker at the event, I struggled to come up with a burning technical issue that I had not yet addressed in previous years. In fact, what I had experienced in the field in the months before (and would in those to come) had little to do with technology. Whenever IoT was hard to implement or didn’t happen at all, there was a strong human component.
For Zühlke I’ve had a unique perspective into the heart of many companies. As an engineering office, Zühlke offers business consulting around the digital transformation, provides support if customers want to learn how to better implement new hard- and software products, or does bespoke development. Especially change management and organisational deployment activities offered plenty of insight. That’s what my thingmonk talk was going to be about…
The complete deck is on SlideShare.
My talk set out with a historical perspective. There are many reasons for the decline of the Roman Empire. One reason that is particularly debated in the book The Collapse of Complex Societies by Joseph A. Tainter is that of its organisational structure. The hypothesis is that because of geographic expansion, the distribution of information and goods required continuous addition of organisational complexity, to a point that the very organisation was no longer sustainable. (I stumbled across the parallels between modern enterprises and the Empire in a rather interesting blog post by think tank Stop.)
The localisation of Roman legions serves well to highlight some of the issues. Their deployment to zones of unrest is a strategic decision, the consequences long term and the flow of information follows a complex command structure. Much like in a large enterprise, it could take weeks or months for crucial information to reach its recipient.
It is not difficult to imagine how miscommunication, a lack of knowledge and suboptimal organisational structure led to the collapse of the Roman Empire and the Tower of Babel, and may contribute to similar problems in modern times.
In fact, whenever I observed bottlenecks and stalling progress in projects, it was because they were cross-functional and required a holistic view of the work, and how said work would have an impact on many different levels of the organisation: IoT, data science, agile development or DevOps — many large organisations struggle with them.
The issues that arise are universal, irrespective of company structure. All large organisations are to some degree hierarchical. The question is how these hierarchies are balanced and how responsibilities are distributed, how communication is established, and if everyone in the organisation understands the common goal.
The key problems that I’ve experienced:
- Unbalanced responsibilities and power play. In one case, it wasn’t possible to get the hardware and embedded software team into the same room except for workshops with us. They were managed by different people, anchored at different levels of the hierarchy, and their collaboration hindered by more pressing issues both teams had to address. In another case, a data analytics project, managers of one and the same organisation behaved rather territorial about their data.
- Process everywhere. Creativity and innovation can only flourish where people are allowed to think and go unusual routes. Going unusual routes is often hard enough. When you throw further obstacles into their way, and be it administrative form R532, on paper, signed and stamped, don’t be surprised if they rather stick to unexciting, conventional ways. This has probably the biggest impact on deploying IoT in the enterprise as well. Suppose a process-driven company wants to introduce IoT to do things a bit more efficiently… …expect resistance from the people whose well-defined and battlefield-proven processes you’re destroying by doing so.
- Large organisation don’t know if or how to eat the elephant. Unless companies are already to a large degree digital, they often have a natural fear of uncontrollable complexity. The thinking goes like this: Yes, we would like to do IoT. But to do IoT, you need to be in the cloud. The cloud means that you do big data. For big data, you need to hire those expensive data scientists, and in order to afford those, you need to have an IoT strategy in place. While this thinking is, of course, nonsense, it plays into the hands of those who do not want to implement a digital strategy. At the same time, the same thinking allows proponents of a particular hype (big data, cloud, data analytics) to push their agenda: Because you need cloud first, no? However, budgets are typically limited, and ultimately, this circular thinking hinders the implementation of the small, pragmatic solutions that are really needed.
With that, I zoomed further into IoT development and looked at team structure.
I defined a team as a set of interdependent players that follow a simple command structure. As the organisation determines strategic movement, the team is involved in short term tactical decision making. What counts in a team is the joint experience of often multidisciplinary players.
Teams are important in IoT, as end-to-end projects require a division of labour. While there are quite a few polymaths in the field who can deliver everything from prototypic PCB to complete solution, tongue in cheek I’ve had to admit that growing your favourite Node-RED flow into a certified, cloud-connected consumer product is quite a different matter. We need specialists coming together and work hand-in-hand. However, team communication often reminds me of the fallacies of distributed computing, but with people.
IoT isn’t going to go away anytime soon, and so aren’t IoT developers. Forecasts from 2014 as to the number of developers in 2020 were already outdated in 2015, and the guestimates are somewhere near the ten million mark. How are we going to organise them?
Most organisation go through organic growth. At the beginning, we are looking at a company where everyone is the team. The team combines the available skills, communication is easy and the common goal is clear. Even if there are several projects, the team exists as one. As the organisation is growing, there are going to be different projects, each requiring a particular set of skills. Some team members are assigned exclusively to a team, others are shared between projects. Communication across the organisation resembles information flow in a losely connected network, with those shared people often taking the role of information hubs. Interestingly, when looking at the various roles of team members in what I called the “collective”, I was reminded of the influential blog post Scaling Agile @ Spotify (link currently defunct, see this comment on what it entails).
Ultimately, at a certain size, the matrix seems inevitable. Here, people belong to pools of a particular skill and projects are staffed by picking the required skill into a temporary team. The matrix has the best properties for scaling an organisation, avoiding unutilised skill by quickly assigning people into a new team where they can perform. That, in a way, holds true for the collective as well. However, due to scale, team assignment in the collective happens much more through volunteering and self-selection, whereas in the matrix the assignment is an administrative process. If work is a happy place and every project is exciting, collective or matrix doesn’t matter. However, if things are difficult, on a psychological level taking owership of a project is much more difficult when being staffed into such project than deliberately chosing a challenging task. The matrix has a further, fundamental flaw: By it’s very logic, people of different skills are organised into intellectual silos, making interdisciplinary thinking outside concrete projects a lot more difficult.
There are good reasons for matrix organisation. It is efficient. Having had a career as academic group leader, I’m aware of the time and effort that goes into organising researchers into project teams so that everyone is happy. And a company isn’t a Montessori school. However, there are other examples of self-selection that were successful, most notably within the RAF that used it as a strategy to cast Lancaster bomber crews. (See also this really interesting blog post on self-selection in software).
One reason for self-selection and longer term teams is the emotional effort and efficiency loss that goes into forming teams. Research that is more than 50 years old (by Bruce Tuckman) shows that teams work best when everyone knows their place, is aware of the team’s expectations and shares the common goal. The main goal is to minimise friction loss, and good communication, processes and shared rituals are key. However, if a person is constantly going through the storming phase because they have been assigned to a short term team by decree, this can potentially jeopardise said team’s productivity if not every other team the person belongs to.
On my hunt for information how large organisations deal with forming and maintaining high-performance teams, I stumbled across US Army publication FM 6–0. It seems the military is well aware of the issues and how to balance “human and technical dimensions”. Another aspect that affects the military is that of functional redundancy: They tend to avoid specialisation to a degree where only the one guy in the team can throw a grenade. Up to the platoon level with 30 or so members (or three organisational hierarchies), people are generally trained to do their comrades’ job if need be.
Compare that to the superhero, the one person who needs to be staffed into every project just because she knows how to set up a DevOps environment for the others, or the only developer who really understands the architecture of a solution.
I was rather pleased that the bus factor still warranted a mention.
Just adding more people into a project alone and sharing skills between them doesn’t help in IoT or other cross-functional topics. Creating true interdisciplinary teams requires more. It is the shared understanding of the challenges, a deep appreciation of the other team members’ roles and a common, shared vocabulary to talk about certain aspects of the work in an articulated, detailed manner. I exemplified this on the term latency and what it means to different people in an end-to-end IoT solution. An embedded systems programmer may think of a latency as good if her hardware can react to a trigger within a 50 microsecond time scale, whereas someone orchestrating containers in a cloud backend may be used to time frames of minutes, and good latency for a frontend designer is everything that doesn’t annoy the user. They all talk about the same concept, but the context and scale is different for each of them. Other terms meaning different things to different developers are architecture and security.
The basic understanding of all aspects of an IoT solution as well as the need for a common vocabulary that allows specialists from different backgrounds to talk about IoT was the main reason for my book The Technical Foundations of IoT (Artech House, Boston).
I’ve been criticised for pushing for so much academic rigour into IoT, as in many cases it doesn’t matter if you truly understand the basics of Internet routing or the physical constraints of radio signals to deliver a solution.
It is this deep knowledge though that fosters true interdisciplinary understanding of a topic. While thinking that your everyday WiFi signal has to go through a Fresnel zone or that it is really hard to guarantee a message has been received exactly once doesn’t add value to your IoT solution, “working out funny relations and interesting things” prepares us better for situations when our solutions don’t behave as expected.
This wide range of experiences, skills and vocabulary is nowadays often reflected in job ads aiming to find the T-, F- or pi-shaped engineer. Rather than looking for a very specialised, super focussed engineer, there is a growing appreciation of people who are jacks of all trades, who know how to speak the language of other specialities. The T stands for one specialisation, but broad technical understanding. The F often represents broad knowledge in several technologies (say, IoT and data science), and the pi-shaped engineer has experience in the application of her skills in several verticals.
Good communication and the ability to talk about work is the key to agile development, and good project teams use these generalists as social and technological glue.
Does building an IoT solution require a scientific mind or is it an engineering process? I argue it’s both. I often see engineers bending a problem until it fits their methodological toolbox, the duct tape in the picture, so to speak. And from Yodit Stanton’s presentation 10 Reasons Your IoT Project Will Fail it becomes clear how common it is that unsuitable sensors are being used just because there is no appreciation for experimental data:
Also, IoT is about the network of things, and networks follow complex behaviour and a prone to emerging properties. This is nothing engineers are really trained for, but scientists are. Hence, there is no shame in coming to IoT from a scientific rather than an engineering background.
I concluded with:
We can’t change organisations, but we have to be aware how organisational structure can hinder innovation and work against us. And good project teams are hard to build, but a lot easier to maintain with the right people. Self-selection can help. And value jacks of all trades!
