[something] DevOps
Yesterday, at the #watercooler, we were talking about the principals of DevOps (and how they were applied before the term was cool) and sharing war stories about organizations that preferred to intentionally wall off individual technology groups. Conversation turned to how running the code in production is the only measurable way to validate developers code, and instilling the culture that shares the responsibility for production readiness between operations and development teams is the way to go. As we were talking about it, I realized that despite the fact that majority of developers firmly believe that “It worked on my laptop” is a piss poor excuse for production failures, most don’t truly understand why it is virtually impossible to make your development environment representative of production.
The primary reason everyone mention, when asked, is a technology stack spec/configuration differences. While it’s true, thanks to the black magic of Cloud (capitalization intended) with a bit of wizardry from Chef one can create a pretty reliable replica of the production environment on demand. The actual main issue with reliable production mirroring is complex, but can be described in one word — data.
Quantity of data
Let’s be honest, most of the time developers don’t have the full dataset to work with in their development environment. And testing application against 10 row table vs 10,000,000 may produce significantly different results. As a most basic example, N+1 problem will not be noticeable on the former, but will bring your production to its knees with the latter. And even if a developer decides to be diligent and attempt to recreate full production data store into their development environment, the data will be out of sync as soon as import is finished. And with developers luck, in accordance with Murphy’s Law, it’s the 10,000,001st record that will be the straw that breaks the back of your application in production.
Quality of data
Every user is … special. Production data helps to find edge cases that would’ve not have been thought about even in the wildest developer [dream|nightmare]. If you’re testing a form with a name field, chances are you’ll test with “test test” or your own name. Same chances would suggest that your name is not “Geschwindigkeitsüberschreitung Füße”. Make your peace with this one. You can never reproduce every production data scenario in your development environment. Ever. No matter how many data validators you write or how many test suites you create, you will not think about accounting for Wolfe+585, Sr. in the said name field.
Frequency of data
This part actually reinforces the two points above — scale and predictability. Test results by a single developer, in isolated development environment, will not provide an adequate representation of the same functionality with 10,000 concurrent connections. Similarly, even if you devise a load testing suite to test for predictable traffic patterns, you cannot account for the unknown. As a very real example, hacker’s brute force attack on your password protected admin section can, best case scenario, lock up authentication for all your users. Worst case — your production is back on its knees.
And because (for all the reasons) development will never truly represent production, identifying and troubleshooting issues in production becomes critical for companies. But it order to do that, developers need to have access to said production (if you care about time-to-solution that is). Give it to them. Call it DevOps if you want. Just don’t give your developers “[something] DevOps” title. Ever. Please.
