A Comprehensive Guide to Securing your Infrastructure

By Kevin Cherian, Senior Staff Software Engineer at Eightfold.ai

At Eightfold.ai our mission is to enable the right career for everyone in the world. We strongly believe that people are the biggest contributor to an organization’s success. Over the past year, as Eightfold.ai has scaled to support some of the largest organizations in the world, a highly secure infrastructure has become of paramount importance. This blog hopes to become a resource for teams looking to build infrastructure with security in mind.

Our framework for thinking about security of our infrastructure is the following:

1. Securing our application (product)
2. Securing our cloud and network infrastructure
3. Continuous monitoring and security upgrades
4. Compliance & reporting

Securing our Application

We have spent time making improvements to our services to address common security issues that many product frameworks do not address out of the box.

Secure Access

In our platform, we ensure that no unencrypted connections are allowed, and all HTTP connections, if any, are redirected to use secure HTTPS. We disable deprecated TLS versions and require TLS 1.2, with forward secrecy, and only use strong cryptographic algorithms. If you use a security policy that employs forward secrecy, for every user session, a new unique key is generated, which ensures that a single session key being compromised does not allow an attacker to decrypt data exchanged in past sessions. SSL Labs by Qualys is a popular SSL testing tool to check latest vulnerabilities and misconfiguration.

SSL Report for our application:

Secure Authentication

We support a few different ways to authenticate users with security in mind:

1. SSO: Single Sign-On allows organizations to fully manage their authentication process. This reduces the proliferation of passwords across different services a user might use. Together with strong password policies and MFA (multi-factor authentication), SSO serves as a robust and secure way of authenticating your users. SSO is the preferred method of authentication for enterprise access to the Eightfold platform.
2. Passwords: When authenticating via passwords, we require strong password policies, strong session timeouts and most importantly, only store passwords using secure cryptographic algorithms. When choosing a password hashing function, opt for one that is appropriately slow for an attacker and resistant to brute force attacks while not being unusable for honest users.

Cross Site Request Forgery attacks

Cross Site Request Forgery (CSRF) can be described as an attack that takes advantage of the fact that a user is already authenticated into your application and the attacker uses that state (being the application cookie) to perform requests the user didn’t intend to on your site. Let’s take the following example to explain this attack a little better. If you are an admin user in the Eightfold product, you have the ability to control which users in your organization have access to the product. Now, let’s say an admin user is logged in and visits another website that has a URL that when clicked, fires requests to the Eightfold product to remove access for all users in your organization. The browser sends the same cookie used to manage the state that you are logged in to Eightfold in the request to remove access. If your framework has not protected against CSRF attacks, the admin user would have inadvertently revoked access for all users in the organization by clicking on a link on a different website. To protect against attacks like this, we’ve implemented support in our application framework to inject CSRF tokens as part of requests. We use the current user session information, a secret key and a salt to ensure that this token is unique. We validate this token to ensure that this token has not exceeded a specified token expiry time.

Man in the middle attacks and HTTP Strict Transport Security (HSTS)

A man in the middle attack (MITM) is an attack in which a malicious agent intercepts and modifies communication between two parties. To protect against these classes of attacks, we enabled the Strict Transport Security header. This ensures that traffic to our application only goes through HTTPS.

X-Frame-Options

We have solved for a class of attacks known as “clickjacking” attacks. A “clickjacking” attack is typically implemented as a transparent layer on top of your product’s original page and when the user tries to click on the original page’s content, they unintentionally execute code sitting in the transparent layer. To prevent this class of attacks, we set a strict X-Frame-Options header in all our responses, thereby ensuring that our website would not be embedded as an I-Frame on external sites. Wikipedia does a great job of listing past clickjacking attacks and explaining clickjacking in more detail.

Content Security Policy and Cache Control Headers

Content Security Policy (CSP) is a layer of security that allows browsers to protect users from XSS and packet sniffing attacks. We added measures to prevent XSS attacks by setting X-Content-Type-Options header and forcing the browser to no longer detect response content type. We set our content security policy to only serve content from our domains. Additionally, we disabled HTTP and browser caching to avoid private information accidentally leaking.

Secrets — Passwords, API Keys, Certificates

Any passwords, API keys, authentication tokens, hashing salts that allow secure access to systems should be considered a secret. Secrets should not live in your code base or in an unencrypted form. We secure the following secrets by storing them to a Secrets Manager:

• Passwords (Database, Search Platform etc)
• Access Keys
• Certificates
• API Keys for external services
• SSH keys
• Hashing salts

In addition to ensuring your secrets are encrypted at rest and accessed via secure method, it is also important to periodically rotate secrets. We rotate these secrets at a predefined frequency with ZERO downtime of our platform.

Web Application Firewall

As part of our effort to enable continuous protection against application level threats, we have a web application firewall (WAF). A WAF acts as a safeguard against common security threats, by blocking dangerous traffic before it reaches your application. At Eightfold.ai, we’ve defined a set of access control rules that every application load balancer uses to determine what traffic is allowed to proceed forward to the application. These rules protect against a variety of things from malicious bots, SQL Injection, Cross Site Scripting to application specific rules.

Static Code Analysis

We also make use of static code analysis tools to identify common vulnerabilities in our codebase as part of our development and deployment process.

Securing our Cloud and Network Infrastructure

Broadly, we think of securing our infrastructure in two buckets: securing our data stores and securing our core infrastructure.

Encryption of Data at Rest & In-Transit

To secure our data stores, we made security considerations around our primary and secondary data storage systems. These include:

Databases

• We use databases as our primary data store at Eightfold. We encrypt our database instances, their associated snapshots at rest and in-transit.

Distributed File Storage System

• We follow best practices with our DFS, encrypted data at rest. Access in-transit is encrypted.

Data Warehouses

• We use a data warehouse at Eightfold to drive our analytics applications and logging infrastructure. Data here is encrypted at rest and in-transit.

Caching/In-Memory data stores

• We use an in-memory data store to power our caching workload. All data here was encrypted and in-transit.

Search Platform & Microservices

• At Eightfold, we make use of a search platform to power several of our products that require powerful search and fast data retrieval. To secure this system, we enabled authentication for requests to our search service, enabled encryption at rest and in-transit.

Role Based Access Control

We implemented Role Based Access Control (RBAC) with mandatory MFA, ensuring that all users only have the minimum access they need to perform their role. Our infrastructure operations team reviews the set of permissions each user has on a frequent interval and revokes any access that’s not needed to perform their role.

CIS Hardening

All instances that we operate in our production infrastructure are hardened to meet the CIS (Center for Internet Security) benchmarks. CIS Benchmarksare a globally recognized security standard for infrastructure. We hardened all of our production images to close unnecessary ports, limit access to only the root user on several system level directories, enforce password strength, and implemented several other security improvements in accordance with the CIS benchmarks. We run these updates at regular intervals to ensure that our infrastructure is always compliant with these standards.

FIPS 140–2

FIPS (Federal Information Processing Standards) 140–2 defines requirements and benchmarks around effectiveness of cryptographic algorithms. We enable FIPS 140–2 standard encryption around several of our services whenever encryption modules are being used.

Network Security

In addition to the above, we continuously audit network access control settings of all our systems to ensure that access is tightly controlled from whitelisted IP sources and disabled for all public access.

Another step we took to secure our infrastructure is enabling VPN with multi-factor authentication in every region that we support. All access to our private network is only via VPN providing strict security and audit capabilities.

Continuous Monitoring and Security Upgrades

Security is not a one-time effort, but a sustained one. To enable this, we have tools to allow us to perform continuous kernel upgrades, scan our network for threats and audit all activity within our infrastructure.

Continuous Kernel Upgrades

We have automated playbooks to enable continuous kernel and package upgrades. This is run at a periodic frequency to ensure that all instances are up-to-date and the playbook determines whether any updates require reboots to take into effect. Our entire infrastructure is built in a way to tolerate a small number of machine restarts and this allows us to seamlessly issue staggered reboots of the entire infrastructure in a safe manner.

Continuous Network Scanning and Activity Auditing

We make use of vulnerability management tools to identify and categorize security risks in our infrastructure. We store logs related to account activity in our infrastructure and use these logs to enable our operational audit processes and to increase visibility into possible security vulnerabilities. Additionally, we make use of web application scanners to discover and catalog security issues in our application.

Compliance and Reporting

As your company grows and scales to support the largest enterprises in the world, compliance with a few common security frameworks is essential. These frameworks include SOC 2 Type I and Type II, NIST, and others. These frameworks provide a standard way for your company to design controls and standards in your infrastructure around security.

As your company scales to support large enterprises, compliance with these frameworks becomes integral. If your company intends to build products in the cloud that can be used by the Federal Government of the United States, you’d need to achieve compliance with a program called FedRAMP, which is the government’s most rigorous security compliance framework.

At Eightfold, we make use of controls in these frameworks to guide our process around managing customer data and ensuring we meet global standards for security and integrity.

We hope this blog acts as a framework your team can use to approach security at your organization. We would also like to share a link to our security bug bounty program that helps us continuously improve security of our entire product.