Falling into the pit of success
At Alan, we want our software engineers to focus on building delightful features for our members, not fixing their dev environment or wondering if their change will break production.
In a word: efficiency, but efficiency that scales. We want no friction, or friction that nudges you in the right direction.
If you are an avid reader of our blog, you probably already read our various articles on enforcing or sharing good practices. Here we want to expand this notion by talking about “THE PIT OF SUCCESS” (imagine the sound of rolling thunder while reading this).
First, we didn’t invent it. It was invented by Rico Mariani around 2003. His definition is:
The Pit of Success: in stark contrast to a summit, a peak, or a journey across a desert to find victory through many trials and surprises, we want our customers to simply fall into winning practices by using our platform and frameworks.
Internally, we rephrased his definition by: It must be easy to do the right thing, and hard to do the bad thing
This motto is one of the drivers of our developer experience team, and most of our company in fact.
Let’s see a few examples of how we implemented this.
Making sure we know what’s wrong and showing how to fix it
We recently migrated to expo for our mobile apps. One of the most common errors experienced by our fellow engineers was an outdated dev client. Instead of just failing, we implemented a pop up that tells what’s the problem and how to fix it. You don’t need to be an expo expert to understand and fix the issue now!
Building this pop up took 2–3 hours from one engineer. This is clearly nothing compared to the time to get/ask the solution on slack/google by our 90 engineers.
Making sure the code doesn’t do weird things behind our back
We use a relational database because we need consistency. This implies, sometimes, to carefully manage transactions and when we do a `commit` into the database. But it can be hard to know if a function you call doesn’t already do a commit, or changes in the transaction. That’s why we added a python context manager called `no_commit_in_session` that prevents all commits and rollbacks within a block of code.
We use it heavily in our claim engine.
Making sure we don’t waste time
We have a heavy use of the tool pre-commit. This allows us to run some checks before even committing, hence speeding our development cycle. For example, we run our linters and formatters in a pre-commit, so we don’t need to wait for the CI to fail.
We also use it to prevent absent-minded engineers, like myself, from committing to our main git branch. This allows us to make sure we don’t make a mistake, but we can also bypass it if we want.
Making sure we don’t do a (stupid) mistake
Our application sends emails. And it’s quite useful for engineers to test this locally. So we implemented a check that makes sure you can’t send external emails to random people. Our mailer, in dev, can only send emails to our domain `alan.com`. So no-one at `example.org` receives email from us!
Another good example is commands. We heavily use flask commands. Some were made only to run locally or in staging. To prevent any mistakes we added a decorator `@do_not_run_in_prod` that makes the command fail if it’s run in … production.
Making sure we didn’t forget something
We have a lot of scheduled tasks in our backend. A scheduled task is composed of a python function and a line in a YAML file. To make sure we don’t have a typo, we added a unit test that checks this YAML file and fails if any of the commands doesn’t exist. We added it after pulling out hairs on trying to understand why a command was not run. There was a simple typo in the YAML and it took us hours to spot it.
Conclusion
We agree that, taken separately, none of those examples are groundbreaking. It’s just the addition of all of them that makes it interesting. For us, “Falling into the pit of success” is not sometimes, it’s baked into our engineering principles. It’s so present that we have a slack emoji for it, and it even starts to spread into other parts of the company.
Your turn, what are the other ways to “fall into the pit of success”?
