Paradoxes of DevSecOps at the Seed Round

At the seed funding stage, startups face a unique set of challenges, one of which involves the integration of security into the development and operations workflow — DevSecOps. This integration presents paradoxical situations where the need for rapid development and the implementation of effective security measures often conflict. Startups are driven by the urgency to innovate and scale efficiently, yet they must also establish stringent security protocols to protect their burgeoning digital assets. This delicate balance becomes critical during the seed round, where the expectations for quick delivery can clash with the essential requirements for robust security. We delve into these paradoxes and offer insights into how startups can navigate the complexities of DevSecOps without compromising on security or speed.

Security is an Antipattern

If you join a seed round startup in any type of infrastructure role you will probably be in charge of everything security-related in addition to your core responsibilities. You will be responsible for compliance and making sure your new company doesn’t have its database exfiltrated to the dark web.

Security seems like a no-brainer to implement across the organization, but it wouldn’t be long until you face backlash. Security is often seen as compliance and nothing more. Anything more than compliance is seen as detracting value from the business.

There is a tradeoff between productivity and security. The more secure you make an information system, the more cumbersome the system is to develop for or use. Making the software more secure and shifting left by adding code scanning and SBOMs deeper into the Software Development Lifecycle usually annoys the developers more than anything else.

Companies want to say they have significant cybersecurity policies and controls because they want their customers to trust doing business with them. After trust and compliance, most leadership at small companies do not want any additional investments in cybersecurity.

Cybersecurity is about risk mitigation or reducing the overall likelihood of risk of a cyber breach. Starting a seed round startup is already a high risk, high reward endeavor, so a little bit of additional risk is already small. Most startups fail, and cash is usually allocated to the revenue-generating pillars of the company.

Adding pre-commit hooks to look for code smells and security vulnerabilities will be seen as a waste of time. Depending on the leadership of your company, you may even encounter ethical dilemmas or more grey areas when executives would rather lie about compliance than actually comply with GDPR or other privacy laws.

If you want to focus on the security of the SaaS solution, it is better to join the company in Series A or later and try to mitigate all security shortcuts that were taken earlier.

Leverage of Technical Debt

When it comes to creating new software companies, I see two trains of thought when it comes to building software:

• Deploy and build as soon as possible
• Focus on architecture and design before shipping code

The first method focuses on getting the code to customers as quickly as possible. You are joining a phase of the company that is potentially pre-revenue or, at the very least, bleeding cash every month. Executives want to provide something valuable to customers that will generate revenue and eventually lead the company to profitability. Spending time deploying features is preferred to excessive time spent on code reviews and architecture diagrams that will eventually change.

The second method involves slow, methodical code reviews focused on building things right the first time. Making code that design decisions that are easy to scale, maintain, and test. Reducing technical debt allows the business to lower overall development time on the product for the entire lifecycle of the product over years, but with an additional upfront cost of design. Code quality is preferred over speed of development.

So, what is the right choice for your business?

Probably a mixture of the two. I’ve seen large enterprises spend years on engagements that should have taken three weeks, nitpicking code design choices. This is not time efficient, and the technical debt cost to the organization is offset by the additional time in code reviews. On the other hand, writing poor quality code reduces stability, testability, and the long- term productivity of software development.

In finance, debt is not always seen as a bad thing. There is good debt and bad debt. Good debt could be a small loan for a factory to increase production of your business or taking out a student loan for an engineering degree. Bad debt would be buying hundreds of video games on a credit card for personal entertainment. Debt is a powerful instrument, and in accounting, there are different types of analysis called leverage that look at debt in the business and the overall health of the organization’s finances.

Taking on debt to increase overall productivity is okay as long as the debt is not excessive and you have the cash to pay off the loan over time. Technical debt can be looked at the same way as financial debt. The product manager usually prioritizes shipping Products, and the Software Architect usually prioritizes design, but you can do both.

In the seed round, the business is trying to show product-market fit. Scalability is planned in the future but at the moment speed of delivery of new features is what is desired. Writing a little bit of dirty or unclean code is a tradeoff worth making if it means a faster time to market. Getting beta customers and improving customer satisfaction are required for additional funding.

You will be refactoring most of the code base in Series A or B. This doesn’t mean developers shouldn’t write clean code when possible, but it does mean the engineers should balance the needs of the business and know that shipping code is more important in the short term and long term. Most of the code will be rewritten when additional funding is available.

Visibility and Observability

Being an SRE, platform engineer, devops engineer, or some other related role usually entails responsibility for observability. Knowing what metrics to monitor, logs to filter, and alerts to triage is as much an art as it is a science. These are hard problems for an enterprise and even harder for a startup.

As a cloud professional, you will probably choose a paid product for a log aggregation platform. AWS CloudWatch or Azure Monitor are not granular enough with features for the application monitoring platform.

The best tools in this space are also the most expensive and it is hard to get buy-in from management and persuade management that you need a particular tool. Most log aggregation platforms historically had a cost structure that was based on the amount of data ingested each day. On a lean budget that meant the startup often limits the total amount of data ingested each day to manage the budget. The side effect is that when when something goes wrong you do not see the logs or metrics in the log aggregation platform because the system had been throttling data from the past 4 hours.

Too many alerts do not increase stability but instead decrease visibility by sending too many alerts to the SRE. Engineers will end up muting alerts and stop paying attention, leading to more true positives being ignored along with the false negatives.

Engineers also can’t have alerts from all sources but will have to pick and choose. AWS VPC flows take up a disproportionate amount of GB of ingest for standard Kubernetes clusters based on normal traffic and the data being ingested. You will have to cut out sources and keep tweaking the alerts and data sources to be within your budget and still provide visibility.

To make troubleshooting even more complicated, when an application with a health check fails in Kubernetes the application will often spam the log aggeration platform as the pods restart over and over again from a failed healthcheck. Quickly, your log aggregation platform will be throttled, and you will not receive any alerts or metrics. Now you are effectively flying blind and you will need to pay more money wait until the next business day to deploy any new features and hope the customer doesn’t experience any bugs.

Again, your security operations will be a secondary priority. You will create several alerts for security events that you will want to monitor, such as logs in your Web Application Firewall, but these will always be seen as lower priority compared to other business priorities.

You will be expected to be on call more than your coworkers until additional funding occurs. Knowing the health of the application, what to measure, and what security incidents to look at is one of the most important aspects of your role for the organization.

Measuring DevSecOps

The hardest thing about DevOps is that most executives and even technical managers don’t know or disagree on what DevOps is. Some people think you're a software engineer, some think you're just a system admin who clicks buttons in a cloud console, and others don’t comprehend the term enough to even form an opinion.

Leadership is excited about a new feature, a UI design makeover, or some other functionality they understand. DevOps is about making the Developers more efficient at their jobs and often is responsible for nonfunctional requirements. These include the latency a web page takes to load, optimizing performance, and autoscaling the backend. Product managers think infrastructure personnel don’t know the business, which is true to some extent because the DevOps professional is not designing the UI workflow or looking at the user stories that the customers look at.

It’s not that SREs don’t know the business; it’s that we, infrastructure people, are looking at a different part of the business than most of the employees are looking at. Infrastructure automation is a jargon-filled profession, and this position often comes across as the absent-minded geek who is more obsessed with Linux system administration than the pricing model of your company's product. I think most infrastructure personnel may be interested in the pricing of the product, but the roles and responsibilities of the product are large enough in scope that they to be an expert in both would be too much responsibility for one person. DevOps engineers should leave most product decisions to the product manager, and the product manager should leave most infrastructure scaling automation to the devops engineers.

Business acumen is a very subjective measure. Different organizations operate differently and have different management styles. Being a good business leader at Netflix is very different from being a business strategist at Walmart. DevOps Engineers communicate differently than marketing personnel, and our end customers are ultimately our internal developers.

But this disconnect changes how the DevOps role is viewed. DevOps is viewed as an auxiliary supporting role of the business but not part of the business. A supporting role. Startups need to be lean, and leadership only wants to support revenue-generating parts of the company and downsize cost centers. It is easier for engineers making functional features such as React UI, to see the impact of the value of the different product design features and feel connected to the end product.

Security is only slightly better. Movies and TV shows have given a glorified portrayal of hackers, such as Mr. Robot. The downside, again, is that most executives do not want to invest in security because security is a cost that does not generate revenue. In short, leadership doesn’t want to invest in DevOps because they don’t understand DevOps and don’t want to. Executives don’t want to invest in security because they understand it a little bit more but would rather take a risk than invest additional money.

Your opinions will be pigeonholed into the DevOps and security sector of the business. You will not be in most client-facing Zoom calls. The business did not hire you as a developer or a product manager. You will be expected to stay in your lane, and this expectation will vary widely depending on your manager’s expectations of DevOps.

A mistake I made early in my career was not looking at some of the small cost savings. If I saw the business was being billed for 15 dollars a month for a minor cloud configuration inconvenience that would take 4 hours to solve, I usually thought this was a waste of time. The opportunity cost to my employer, after considering salary, payroll taxes, and health insurance, was over 100 dollars an hour. Five hours of my time cost the business 500 dollars. It would take 34 months to make up for the opportunity cost of my time to equal the cost of the misconfiguration or almost three years to break even on the investment of the misconfiguration and the chances the application would not be rearchitected and designed before three years were low. However, my supervisor appreciated deeply researched reports on the visibility of cost savings on cloud resources and preferred that the decision to change a configuration be the senior management's choice instead of an ad-hoc decision. What if doing additional research for a 15-dollar-month cost was found to increase to 75 dollars a month? Then, the math of the opportunity cost changes. It was a trade-off between decision speed and priorities. Having a strong communication dialogue with your supervisor on their management style and the expectations of your role is paramount to your success.

DevOps is a broad term involving many different skills and abilities. DevOps is usually a senior title, and lucky enough, like me, to start in DevOps early in your career, you are the exception, not the rule. Learn your manager’s expectations of the role and try to exceed them. This also means it will be rare to receive a raise, your work is invisible to the company unless something significant is broken; then you will be out of a job.

Estimating Cloud Cost

Setting the budget for the infrastructure is one of the hardest parts of starting a company. It may seem simple … 5 EC2 M5 instances is .50 cents an hour plus EBS volumes, VPC data migration, and … then the cost keeps adding up from things you didn’t know you were being charged for..

In LastWeekinAWS blogger Corey Quinn writes

“Gone are the days when “instance hours” and “gigabyte months” were the only units of cloud measurement we had to worry about. Now, we grapple with constructs like “I/O Operations” and “LCUs,” which depend on multiple dimensions tied to your choice of Load Balancer. The downside of these obscure metrics is not their complexity per se, but their disconnect from any intuitive understanding of application resource consumption. More often than not, the only way to gauge what a workload will cost on AWS is to deploy it, let it run, and then consult your bill. This is hardly sustainable.”

Modeling and performing simulations of the Sun exploding a simpler feat than accurately estimating a cloud bill with a high degree of accuracy for a complex application that has yet to be developed or deployed to the cloud yet. The cloud providers try to quantify all the costs and depreciation of their hardware into their cloud pricing model. The end result is an extremely complex pricing model that varies widely based on the regions where you are deploying, the architecture, and the latency requirements of the different components of the application.

As an infrastructure engineer, you will likely need to forecast the total price of the infrastructure, say, within the budget. This could involve buying reserved instances, but at the same time, creating the financial forecast is mostly guesswork and will need to be revised and will become another guess every time a major engineering overhaul is complete. You will be expected to estimate the unguessable and lower the cost at all times.

Short Term Projects

The truth of the matter is that the majority of DevOps, site reliability engineering, and platform engineering work is short-term project work. The company will need someone who builds a short-term project to set up a Kubernetes cluster, create the blueprint for CI/CD pipelines. But most experienced developers who are toward the skill ceiling of the profession and would join an early-stage startup already know quite a bit about DevOps. Developers will be able to create their own Terraform, own Github Actions, and their own Kubernetes manifest.

If you are doing the job well, the majority of the job should be fully automated, and a framework for developers should be created for continuous updates.

The infrastructure automation engineer is responsible for creating a platform for the engineers. The devops engineer's job is to make the software engineers' jobs easier. It becomes harder and harder to justify having a dedicated devops engineer at such an early-stage company. The devops engineer needs to be able to fill multiple roles.

Knowing enough software development to be able to do some feature-driven development work would increase the value you bring to the company but remember if you are going to write code, you, as an engineer, need to write high-quality code or stay in your swim lane. At the seed round, there are no junior engineers, and everyone is very good at their jobs.

DevOps engineers are usually not expected to have the same code quality standards as backend engineers, but if you start writing backend code, you will be expected to write code to the same standard. This means knowing how to write reusable, testable, and extensible code that your team will cherish.

I recommend Clean Code by Robert Martin and Refactoring by Kent Beck and Martin Fowler if you are looking for resources to write cleaner code. And, of course, practice, practice, practice! Get a job as a software engineer and practice writing high-quality code in a large organization with strict code review policies. Learning is a struggle, and you need to always grow.

If you want a longer-term job that doesn’t require as much production code, then try consulting. You can move between short-term staff augmentations for years with a consistent workload.

In conclusion, the journey of integrating DevSecOps in the seed round of a startup is fraught with paradoxes that often require navigating a delicate balance between speed and security. This integration serves as a critical idiom in today’s fast-evolving tech landscape, illustrating the need for startups to be agile yet secure, innovative yet mindful of potential vulnerabilities. By understanding and addressing these contradictions, startups can better position themselves to capitalize on their innovations while safeguarding their operations. Ultimately, embracing the complexities of DevSecOps not only enhances the robustness of a startup’s digital infrastructure but also aligns it with the evolving expectations of a dynamic technological ecosystem.