Netflix — Change at the Speed of Innovation — Part 1
Speed Wins!
Netflix is everything about innovating fast and also failing even faster. This is the story of how Netflix is now dominating the movie and media streaming industry.
Netflix has been innovating a lot since it came from a DVD rental business to movie streaming in 2007, expanding internationally in 2010 (only 9 years ago), now with over 137 million users. During this period, Netflix crushed the video rental business and companies like Blockbuster that didn’t embrace innovation at the same speed.
In biology and in business, there’s something called the Red Queen Effect. The Red Queen Effect or Red Queen Race is an evolutionary biology concept inspired by the Through the Looking Glass book (Alice in Wonderland sequel). The Red Queen Effect proposes that organisms must constantly adapt and evolve and not merely reproduce in order to survive. In many populations of organisms, the probability of extinction does not depend on the size and lifetime of the population but on how fast the species can adapt and evolve. The evolution happens when a population uses positive feedback using feedback loops from small changes in the environment.
“Now, here, you see, it takes all the running you can do, to keep in the same place. If you want to get somewhere else, you must run at least twice as fast as that!” The Red Queen from Through the Looking Glass — Lewis Carrol
The same applies to business and, while very traditional businesses like the food industry may be slower to adopt evolve, technology-related businesses must evolve fast with constant feedback loops from the customers and stakeholders. Businesses need to run really fast. If not, they fall behind.
“The rabbit runs faster than the fox, because the rabbit is running for his life while the fox is only running for his dinner.” Aesop
Maybe Blockbuster discovered too late that they were the prey.
What allowed Netflix to innovate so fast and always deliver the best product to the customers? Well, Agile is their second name and “speed at scale” is what they have for breakfast every day!
Netflix is a flat high-performance culture that incentivises everyone to be creative-inventive. A culture based on two pillars: freedom and responsibility.
Mainframe world used to develop really solid software and release a product targeting probably late adopters. Mainframe world and legacy systems know that scale breaks hardware, speed breaks software and speed at scale breaks everything.
So… how did Netflix learned to deliver speed at scale?
Practising failure. It is important to incentivise failure and allow the system to break. This way, not only it allows more innovation inside the organisation but also the team knows what to do when facing failure or incidents.
Netflix is not aiming for perfect solutions. Instead, they are looking for speed and velocity of delivering changes and innovations. It is expected to fail and break but the system is made in a way that failures are easily detected and quickly fixed.
The Netflix Management Model
- Decentralized Decisions — employees have maximum freedom and maximum responsibility
- Fail Fast — A big benefit of fast iteration is to allow to fail fast (and succeed faster). Failing fast allows spending less time doing things that don’t work.
- Agile and Self-Service Deployment — developers are responsible to deploy code but they are also responsible for their code (even at 3 am)
- More Experimentation — The faster the organisation fails, more experimentation and learning it can do. It also removes the fear of innovation and knowledge gaps
- Data Analytics — A lot of data science a big data analytics. Everything is A/B tested and every change is compared and validated through real data
- Learn Quickly — Developers have a platform where they have the freedom to experiment, test and learn quickly
- Culture of Innovation — to be always ready to recognise opportunities as they presented themselves
- Reduced Risk — Fast iterations help to reduce risk during product development. Doing many small changes instead of a few larger ones will mitigate risk.
This management model is possible through a culture of “Freedom and Responsibility" and an infrastructure that allow to easily innovate and deploy in the cloud. This infrastructure consists of a cloud API that basically replaces IT/DevOps, streamlining deployment, testing and rolling back if necessary.
By the way, if we are talking about speed at scale, we need to speak about Knowledge Gaps!
A knowledge gap happens when an organisation tries to develop a product without doing small correction along the way. These corrections aim to make sure that the product is being developed according to the needs of the user. To reduce the knowledge gap is also to reduce the risk of uncertainty and costs.
To make sure the knowledge gap is reduced and the product is being developed according to the “optimal direction”, organisations need to re-target constantly:
- To deploy frequently reduces the cost of correction
- Veer off from the “optimal direction” is expected.
- The ability to work in small batches and get feedback earlier has great advantages over a big batch approach
- Small corrections are easier to make with precision than large corrections
- Small corrections are easier to get approval for implementing from leadership
Web-Scale Architecture
Not only the organisational layer but also the web framework need to allow speed at scale. Netflix uses Cassandra (Apache Cassandra), which is an open source distributed database with peer-to-peer symmetric nodes that replicate several times the data, giving the system a great level of redundancy and making sure that there are no single points of failure (SPoF). Additionally, it has DNS routing and several DNS providers that can be used in case of backup need. AND AWS S3 BUCKED.
Round-the-clock OODA Loop (a very innovative OODA Loop!)
Observe -> Orient -> Decide -> Act loop in a very disruptive, fast and agile way. I’m going to detail it in the second part of this article!
The Continuous Deployment strategy
Netflix practices “Continuous Deployment” SAFE Scaled Agile
- Developers run what they wrote — root access and pagerduty. This way, developers also save the time that they would need to spend to explain to IT od DevOps how to deploy.
- IT is an API = DevOps automation
- Github everything and leverage social coding
- Create a culture to ensure that allows continuous improvement deploying and testing these improvements in minutes or hours.
Some of the advantages of Netflix’s Continuous Innovation Cycle are:
- Test and get feedback in hours or days
- It costs almost nothing
- A smaller knowledge gap
- Cost of failure is very low — easy to rollback
- Easy to deploy
- A/B test everything
- Canary test and canary deployment (test first with a smaller group of users which minimizes the impact of bugs and allow a sandbox environment for testing).
On the next article, I’m going to make a deep dive on different types of OODA Loops and how Netflix adapt it to solve the speed at scale issue!
BTW, take also a look at the 8 Biggest Fintech trends for 2019!
Sources for this article:
http://www.scrumcasestudies.com/netflix-com/
🚀 Follow me and check also my 🧱 blockchain book and course:
