Starting Up Security
Your startup is growing fast. Customers are starting to ask tough questions about security. Your investors and board members have asked to prioritize security projects. The tradeoffs made for early speed and growth are no longer so far ahead of quality and security goals.
But you don’t really know how to “do” this security thing.
Before We Begin
You can’t just hire a couple security engineers to shoulder this burden. You wouldn’t hire anyone to just “go deal with that scale issue” you have either.
Security is part of the overall technical challenge you face, and this article will focus on projects within reach of folks who aren’t familiar.
For those of you reading this from the security industry, prepare to see big topics trivialized more than you’re used to. Startups want to mature from nonexistent security around their proof of concept business, so intimidation via a complicated security roadmap full of buzzwords will be a great way to nurture a breach notification.
So, try not to cringe.
Security Strategy
Let’s look at this in terms of your products, your infrastructure, and your employees and focus on areas of highest impact against risk. We will take big wins over being comprehensive in any specific area.
Your Products
This section will focus mostly on the quality of application development at your company. These are the high level indicators of quality software engineering org that builds security. Great security is a symptom of disciplined engineering, period.
An engineering organization with a few security engineers tacked on is doomed to failure, while an organization of engineers taught to respect security will be far better off.
I would really love to discuss product fraud and abuse issues, but leaving that out for brevity. Sorry. It’s something I like talking about, but not here.
Strict Engineering Standards
Now: You may be working with competing languages and platforms, multiple repositories, inconsistent code reviews and rogue commits. Simplify and standardize engineering processes ASAP. This has immense benefits for every imaginable aspect of quality, not least of which is security. I am a huge fan of how engineering teams use Phabricator here to normalize engineering. Build this into a strong culture of strict code review.
Later: Once some level of sanity is achieved, engineers can build reusable security frameworks that globally handle common issues within your applications, and the review cycle would eliminate exceptions that ignore that hard work. Trigger audits on high risk code for extra review. Treat every security bug as an incident, document it for posterity and push the lessons onward to new and current engineers.
Most importantly, enforce mandatory postmortem of security bugs for leadership, like you would with an outage.
Vulnerability Disclosure and Bug Bounty
Now: Set up a policy that protects security researchers when they disclose vulnerabilities to you, and reward them when they discover security issues with your products. Start your program slowly and invite known researchers at first so as not to be overwhelmed. HackerOne supports this, as do others, but you should totally use HackerOne because of my enormous bias.
Later: Make it an engineering goal to have the most expensive bugs possible. Ramp up your rewards because of an exploitability rareness. Open up to the world and let the hackers hack. Hire your greatest hackers. A healthy bounty program and sufficiently frustrated hackers are a rare quantitative metric of a strong engineering organization with a symptom of security.
Engineer Onboarding
Now: Invest quality time in new engineers and inform them on security sensitive areas of the codebase (Auth, DB, Sessions, Crypto, etc). Teach any homegrown security frameworks. Calibrate code review expectations to the same high standard others will expect. Introduce them to security minded folks and make it OK to reach out for help, and show them where security questions can go. Oh, yeah, build a place for security questions to go. A mailing list, IRC channel, chat room, Google Group, etc.
Most important — destroy a blame culture. Blame the bad guys and anything other than the engineer for a bug, except in cases of severe negligence or apathy. Bugs are not a source of shame and should be shared. Attack failures in process but not the person.
Later: Integrate the history of severe security bugs from version control, bug bounty, and auditor findings into education. Teach, in detail, the tools and tactics your relevant attackers would use against your product. Aspire to be like the Facebook Bootcamp. Have a goal of making it easy for developers to do the right thing and hard to do the wrong thing.
Manage Secrets
Now: Review the crypto around your storage of passwords. Review how many people have access to private keys and certificates. Ensure that infrastructure touching credit cards doesn’t go neglected. Point external audit resource at secure systems with more than PCI compliance in mind. Build services that can make it simple to keep secrets out of source code.
Your Infrastructure
This section pertains to all the systems that support the applications you’ve built. It may include your production, development, build, Q/A, what have you. Bad guys will generally consider these systems their ultimate goal.
Network security is not covered here, as most startups I’ve encountered have it handled through easy cloud provided tools. This is not to say there isn’t plenty you can do here, but we are ruthlessly prioritizing.
Authentication
Now: Multifactor every form of authentication that exists. Duo Security can help protect SSH, and so can Google Authenticator. AWS, or any other cloud hosting platform should all multifactor as well. Lock down root usage to better reflect the activity of individuals on systems. You won’t be able to sort out good from bad if attribution isn’t accurate and everyone looks like the same root user. Ensure that prolific root usage doesn’t become the standard for regular administration and deployment. Root usage should be treated as an extreme anomaly with heavy alerting.
Later: Focus on authorization, “need to know” and least access. If someone doesn’t need access to a whole slew of production systems they’ll never use, then they shouldn’t have access to them. If that someone is compromised, they can’t pivot to literally everything.
Do not let teammates believe this is some lack of trust in one another. Each administrator should already assume they’re compromised nowadays.
This “later” category should all probably be in the “now” category, but sometimes startups are small enough where it doesn’t make sense to prioritize this. Least privilege is more awkward within small groups.
Patching
Now: Your operating systems, kernels, applications, libraries and other dependencies all need to be updated. Software atrophies quickly due to security research. Tools like Nessus can help identify huge holes. Treat patches that fix remotely exploitable bugs with the highest priority.
Most important: When you patch a remotely exploitable bug that was exposed to an attack, make sure it wasn’t actually exploited. (if it was, see Security Breach 101)
Later: Build patch automation within continuous integration. It’s safe to assume that new releases probably fix unannounced security bugs. Remove dependencies that can’t be automatically updated or frequently introduce breaking changes. Minimize this turnaround time to a window of hours.
Keep an eye on Google alerts, Full-Disclosure, and various vulnerability feeds like the National Vulnerability Database for software you’ve deployed. Well maintained software will typically have mailing lists announcing vulnerabilities as well. Regularly schedule vulnerability scanning with enterprise versions of tools like Nessus, Rapid7, etc, with success measured around reduced known vulnerabilities and low windows of exposure between patches.
Centralized Logging
Now: Push system and application logs somewhere centralized and in a separate risk area from the rest of your infrastructure. You’ll need a source of log truth to hedge against a breach. Startups use Papertrail, Loggly, Sumo Logic as examples, but can easily be built in house. Ensure that SSH logs and any internal tools you’ve built log here. Build basic alerting that would tip off engineers when systems are accessed without permission.
Later: Scale this log store and build tools that make review easier or unnecessary. Make these log stores trivially accessible to an incident responder who is working as fast as they can. Make sure alerting happens when logs stop flowing. If an incident can’t be answered quickly with logs — you do not have enough logs. Build log failures into regular incident meetings, as mentioned above.
Your Employees
It’s very rare when an breach occurs and it didn’t start at the employee compromise level. Almost every “sophisticated” breach nowadays begins with a spearphish or waterhole, some sort of zero day, and lateral movement from an employee owned system into a production environment.
Endpoint
Now: Purchase employees laptops to seperate home & work life. Standardize everyone on Chrome configured with Click to Play. Understand and train employees about spear phishing. Enable disk encryption. Build a checklist so employees can self-certify to this standard when they’re hired.
Later: Build laptops with these configurations pre-imaged, or managed via a centralized system like chef. Don’t rely on a employee to-do checklist. Start employing tools like osquery to understand what is being installed across your corporate fleet. Be able to swap out laptops displaying suspicious behavior quickly with new machines to keep employees working.
Shared Passwords
Now: Train employees about the risk of shared password usage between your employees personal lives and corporate accounts. They are likely using the same passwords for their personal email accounts or servers as they are your production infrastructure or corporate email. Pay for Lastpass or 1Password so they can manage the nightmares created by actually secure password management. Push multifactor authentication for personal lives as much as corporate usage.
Later: Systems like Okta, Meldium, Bitium, all help centrally manage credentials for the disparate cloud applications your employees will use. They will also help enforce uniqueness, multifactor, and termination scenarios. Additionally, you can take advantage of SAML (either in house or provided) to centralize authentication as well. OneLogin is an example SAML provider in addition to the above.
Everything Secured
I’ll stop here, because security never really ends. It’s easy for a curmudgeon to look at all the above and say “It’s still not enough! You’re not doing enough!”. And they will. Fuck ‘em. Do your absolute best.
Pro tip: Google for “[Company] not doing enough” security (with the quotes)
The truth is that “enough” comes down to how badly you hate losing to the bad guys. You just have the right focus on high impact security as you grow while finding ways to achieve more. Try to get to later as early as possible.
@magoo
I’m a security guy, former Facebook, Coinbase, and currently an advisor and consultant for a handful of startups. Incident Response and security team building is generally my thing, but I’m mostly all over the place.
