Without automation, what’s the point of coming to work?
Lean TPS, Deployment Automation, and the Path to Employee Wellness
Have you ever liked your job, loved your team members, but somehow things just never achieved their potential? Maybe you all worked 80 hrs a week until your health was jeopardized? Maybe you started to lose faith in people’s ability to get things done anymore? Perhaps the missing ingredient was proper systems automation.
Software systems are organic, empirical animals. Change is constant. Learning is constant. Sometimes our systems seem like a smooth dolphin, moving efficiently, quietly, coordinated. Sometimes our systems feel like a silverback gorilla, completely uncontrollable and scary. Quite obviously, the more uncontrollable, unpredictable and unreliable our systems are, the more our quality of life declines and our team’s success rate suffers. System unpredictability is like living in a soap opera: we make promises we can’t keep, we cheat on our products and we make people very sad.
Lean TPS
Like a lot of people who build software, Lean philosophies have contributed greatly to my end process and products (read my book here). In traditional Lean teachings, we see many references to the Toyota Production System (TPS). With TPS we find a highly orchestrated manufacturing process for repetitive, predictable parts. The basic goal is to generate a perfect balance of internal ‘pull’ and ‘push’ dynamics with orchestrated cooperation between cross-functional employees. We do this in order to accumulate minimum waste and ensure consistent product quality. For reference, ‘pull’ means that an employee decides when to complete a task; ‘push’ means that a task is sent to an employee. From Lean we get an amazing conceptual framework of constraints for production of nearly anything, with ideas like limited batch/queue sizes, limited work in process and minimal cost of ownership.
Because of software’s organic and fluid nature, with each new line of code sometimes changing an entire product, the TPS analogy seems only partially applicable. However, the TPS analogy becomes extremely important when viewed as an important factor for employee wellness. While building software we generally end up with two development components (1) product design/production and (2) integration/deployment.
(1) Design and the early-mid stages of production are largely organic and malleable. Our product vision can see many changes until an idea is ready for execution. Certainly the visual design process goes through many different iterations. In addition, we often perform substantial tech platform research or exploration. Absent some sort of concrete time box, these components of building tend to be very open-ended and unpredictable. Once we have finished with design and early production, we are ready to start ‘shipping’ small components of code to staging/live environments.
(2) In contrast to design and early production cycles, integration and deployment must be rigid, consistent and predictable, like manufacturing. With any software product we want to achieve brutal efficiency and a consistent, safe environment that accommodates constant change, i.e. integrations and/or deployments. The way we approach integration/deployment can dictate or preempt our entire culture around employee wellness.
In the automobile industry, additional cars can be effectively ‘stamped’ out. However in the software industry, we are adding micro-components to the car in realtime while it’s on the road. Elite software systems require consistent creativity and innovation from employees with every deployment. Talented employees need ‘space’ to be creative, while still operating within a framework of tight risk constraints.
From a management perspective, our people want and need a self-managed pull/push structure. This is clearly the only long-term solution for talented people. And so, a very interesting question emerges: How does the company or management get what they want (productivity, innovation and consitent quality), while simultaneously giving its employees what they need (freedom, trust and autonomy)?
Without aligning integration and deployment with some sort of unrelenting manufacturing efficiency analogy, i.e. automation, the space cannot exist to foster self-managed employee freedom.
Lean to the Letter: Code Integration & Deployment Automation
Code integration and deployment are the TPS of software systems. Integration and deployment are exact and perfectly reproducible, save for a few edge cases; they must be 100% automated. Automation at the beginning will always save hours, days, weeks, even months of work over the life of a product. Most importantly, automation empowers people to predictably achieve their goals. Every project I’ve worked on that lacked proper automation for integration and deployment has been subject to pointless headaches, employee heartache and untold failures at every level of magnification.
Integration and deployment automation should include all of these components:
- Code tests — both positive and negative
- Continuous Integration (CI) server
- Auto-deployment to, auto-configuration of, and auto-migration of, development environments
- One-click review/accept for code branches
- System monitoring and reporting (e.g. Nagios, Riemann, etc.)
- Distributed failure handling (e.g. server stops responding)
- Distributed scale handling (this could be optional depending on the application)
Integration and deployment automation sounds like a no-brainer, but I’m still amazed at how many tasks I see performed manually. I would elevate this concern for the Enterprise, where things are downright scary.
With hardcore integration and deployment automation, we remove one of the largest organizational impediments to progress. Once we have finished a design and build cycle, we can approach 100% confidence that there is no impediment to live working software.
Without automation, what’s the point of coming to work?
And so, we circle back to our interesting question: How does the company or management maximize what they want (productivity, innovation and quality), while simultaneously giving employees what they need (freedom, trust and autonomy)?
Without software integration and deployment automation, small problems become large disasters. The currency most affected over time is trust. Lasting businesses do not exist on promises and pitch decks. Real products need to be shipped and updated, consistently. If not, there is an awe-inspiring failure tree smoldering. Executive fails on promise to Investors or Customers. Product Manager fails on promise to Executives. Engineering fails on promise to Product Manager.
What about self-management and freedom?
If we are unable to reliably get software into production, then how can there exist enough trust for employee self-management and freedom? Talented people crave autonomy. Yet, in order to facilitate an environment of trust and autonomy, talented people rely on other talented people, who ultimately rely on system predictability.
I’ve found in practice that a very efficient self-managed structure has limited viability without the right people from the beginning, or without the time and resources to train and educate employees. There must be concrete ‘definitions of done’ with clear expectations of employees, together with an organizational commitment to invest in tooling to achieve expectations. But WHO CARES. All of our sincere intent on self-management is a trifle if we can’t ship code. If code doesn’t arrive to all our environments reliably, i.e. with automation, an entire software organization will underperform and/or fail.
The more reliably we can introduce new code, the more transparency we have, the more trust we build across our organization. Only then can we free up resources to promote employee wellness and empowerment.
The tools we use only make it easier to work in this fashion and to be flexible. […] People here love what they do. Because they love what they do, they want to do the best job. We are trusted here to do our jobs. […] They believe in us. We all believe in each other. I’ve never worked at a place that ever was even close to something like this.
As I’ve grown older, my focus has really shifted away from making money and being ‘successful’ to building a sustainable and compassionate life for myself and the people around me, a life that is rewarding and worth living in the long run.
I’ve found our startup industry to be quite cannibalistic. It seems to eat its young, to sacrifice health and wellness against a mythical construct of what it takes to be successful and ‘killing it.’ I’m quite sure every fast-moving industry has this effect on people. For me this is a problem. We take our most ambitious young people and we poke and prod and rile them up to innovate and get something to market, but at what cost and what does the road look like to get there? To date, I’ve seen our industry focus almost exclusively on Products and Customers, but are we maximizing our Employees’ holistic potential in a sincere way?
There are 100 things I would rather do than deal with organizational impediments. I assume that other experienced people feel the same way. With a disciplined foundation of automation and tooling, we can achieve so many things as a team and start to make more progress toward employee wellness, freedom and empowerment.
Like a talented sports team that combines innate determination with a methodology, some system to reproduce successes, we can get the most out of our people, help them win, keep them happy, and give them the space to be creative where it counts most.
With harmony in the home, the 10,000 things are possible. — Korean Proverb
