Tools for Human, Host, and Infrastructure Security

Uber Security

Follow

Feb 23, 2017 · 6 min read

Chris Gates, senior incident response engineer

This post was adapted from a talk presented at the 2017 RSA Conference. The entire presentation deck is available here: https://www.slideshare.net/chrisgates/devoops-attacks-and-defenses-for-devops-toolchains

Engineers at almost every company use a variety of DevOps tools to increase speed, agility, and ease of their development process. DevOps tools have allowed developers to bypass the typical company infrastructure bureaucracy and develop at the speed they can write code. Unfortunately, many of these tools frequently categorize speed over security or expect implementers of the technology to RTFM to understand any security decisions made by the tool’s creators.

This can create challenges for security teams charged with protecting corporate networks and sensitive data. Do you understand the authorization model of the devops tools used at your company? Are your engineers running out-of-date or insecure implementations? Does one lost access token equal disaster for you company? Below, we outline three areas where security teams can exercise their leverage to harden the developer environment: human, host, and infrastructure security.

Human Security: Employee Intelligence

A primary goal of any modern security team is to make it difficult for employees to allow attackers to walk into the environment. There are a number of ways employees can accidently release data to attackers using external services like Pastebin-like sites, public GitHub gists and code repositories, BitBucket, CodeCommit, etc. There a lot of bots (and humans) that scrape these sites for tokens and other credentials.

Fortunately, there are also a number of defenses available to security teams:

• Move away from public GitHub and replace it with internal Gitlab, Gitolite, GitHub Enterprise, or Phabricator.
• If you must use public GitHub have your team create organization accounts and not just join personal accounts to your organization.
• Audit who has access to your repos and put a process in place for removing employees when they leave the company. Audit personal repos for leaks and regularly search company repos for sensitive data. The open source tool Gitrob can be a good starting point for this.
• Enable two-factor authentication on everything you can.
• Host an internal Pastebin for employees to share code.
• Make use of open source tools for monitoring Pastebin for your organization’s data such as Dump Monitor, pastemon, or pystemon.

Host Security: Workstation Protection

Modern security teams also take responsibility for protecting and monitoring employees on their development workstations (and servers). Developers often have entire code bases on their systems and the ability to create infrastructure for the company via cloud services. They typically store sensitive information (code/tokens/keys/secrets) and are almost always admin on these systems. Security investments in host security go a long way in identifying badness as soon as possible since we can’t always rely on system owners to self-report or even recognize anything nefarious is happening.

Security teams have access to an entire collection of tools for hardening these machines including (not a comprehensive list):

• Osquery is open source and allows you to write SQL queries to explore operating system data for OSX, Linux, and Windows. With osquery, SQL tables represent abstract concepts such as running processes, loaded kernel modules, open network connections, browser plugins, hardware events or file hashes.

• Doorman is an osquery fleet manager for administrators to remotely manage osquery configurations retrieved by nodes.

• BlockBlock is a kernel hook that detects and prompts you whenever software wants to persist.

• Little Snitch is a personal firewall for OS X.

• Carbon Black is a host based agent that monitors process create, writes, registry queries, and net connections. You can create rules and watchlists for known bad behavior.

All the information you collect from various logging sources needs to go somewhere. Stream Alert is an open source tool from Airbnb that provides real-time data analysis framework with point-in-time alerting.

Infrastructure Security: Protect Production

It can be dangerous for developers to build infrastructure without adequate security strategy and training. Therefore, it is in the best interest of security teams to enable developers to build and manage systems and infrastructure with strong authentication and other security best practices.

Continuous Integration (CI)

CI tools like Hudson and Jenkins are very popular among developers, but they have a steady history of security issues. For example, multiple remote code execution (RCE) vulnerabilities have been found in Jenkins over the past several years, advisories are not well publicized, it has weak coverage with vulnerability scanners, and API tokens grant the same access as passwords. At the same time, Jenkins builds and deploys code so compromises can easily cascade into production.

To protect production systems, security teams should consider the following baseline:

• Require authentication on everything they can.
• Vigilantly monitor for security issues and updates for CI tools. Expect the full impact of issues to be watered down in advisories.
• Segment Jenkins from corporate infrastructure
• Logical separation by groups either on single instance or multiple servers
• Monitor Jenkins slave activity and net connections. Osquery is good for this.

In-Memory Databases

In-memory databases such as Redis and Memcache are used as a database, cache and message broker. Redis lacks encrypted communication, credentials by default, secure root access controls, and it binds to all interfaces. Security guidance from Redis developers is to keep it off the internet. Details on how open redis servers are actively exploited are available in the talk slides here.

Memcache is an open source, high-performance, distributed memory object caching system. There is no code execution, but sensitive information frequently finds its way into memcache.

Protecting production systems from vulnerabilities in these databases requires:

• Strong authentication (especially for Redis)
• Binding to localhost if possible
• SSL/TLS if possible
• Isolate in-memory databases from Corp (and the public in general)
• Accurate inventory of the data you put in these databases
• Don’t store keys, passwords, etc.
• Logging to look for abnormal activity. Osquery can be used for this.

What’s Old is New Again

Most of the vulnerabilities discussed will be familiar to industry veterans. We seen the same things for years, just in different environments as developers now deploy and manage the full stack for their applications. As security engineers, we are already equipped to educate them on how to do this securely. Developers have the keys of the kingdom — — it’s on us to protect and monitor those assets.