Secure internet access for applications hosted on container platforms

With the increasing adoption on container technologies, many enterprises are building their infrastructure to support container management platforms. In this blog I will be sharing how DBS overcame internet access restrictions from container-based technology due to its dynamic IP nature.

But first things first: What is container technology?

Containerisation is the process to package an application with necessary dependencies or other required files into a single image. This helps the developers move their application from development to production phase easier and consistently across different environments. Containers are also lightweight, and multiple containers can share the same machines OS system kernel.

To complement the advantages of container technology, the container management platform orchestrates the container runtimes as well as help lighten developers’ toils by automating some of the deployment process.

Challenges faced by application hosted in container platform when using traditional firewall to control internet access.

Kubernetes is a popular container platform and the smallest deployable unit is a ‘pod’ that run with at least one container. Kubernetes improves application reliability through the following:

1. Enhanced scalability of container application by setting several replicas on a specific deployment configuration.
2. Load balancing across replicas via the use of ‘Services’.
3. Auto-healing where the pods will automatically be respawned when it went down due to various reasons such as out of memory.

When an application is containerised and managed by the container management platform, the application pod outgoing IP address will be determined by the node that the pod is hosted on. When a pod is being created or re-created within the cluster due to auto-healing or scaling, the pod may be re-created in another different node which indirectly changes the outgoing IP address.

Pods from different applications can also be created within the same node resulting in different applications sharing the same outgoing IP address.

However, such behaviour means the outgoing IP addresses will be assigned dynamically. This makes traditional firewall controls concept of using source and destination IP addresses ineffective as firewall rules no longer can accurately identify which traffic should be blocked or allowed.

For example, containers of multiple application could be scheduled in the same node as shown in the diagram below. ‘App1’ pods that exits in server 4 and server 5 are supposed to access www.google.com. The firewall rules will allow the IP addresses of these two servers to access the endpoint. This will also create an undesirable situation that also allows ‘App 2’ that collocate in the same node (Server 4) as ‘App 1’ to access the endpoint too.

Some of the existing workarounds these challenges, which are not ideal, are:

1. Whitelist the entire range of IP addresses allocated to the container platform. However, this means that all pods in the platform that are within the same IP address range will share the same access to the internet regardless of the firewall policies defined.
2. Assign static IP for namespace that requires internet access. However, IP addresses are a limiting factor. When more applications are onboarded to the platform, there will eventually be insufficient IP addresses to assign. In addition, the namespace static IP address may be re-assigned to another namespace which may create unnecessary backdoor if not done properly.

Such restrictions in these workarounds formed the key drivers for DBS to create a proxy-based solution Gateway-as-a-Service (GaaS) to support internet access for container platform without compromising security needs.

What is Gateway-as-a-Service (GaaS)?

Gateway-as-a-Service (GaaS) is a proxy appliance that sits between the enterprise systems and the internet that acts as a gateway to control all traffic going in and out of the organisation. It acts as the corporate gatekeeper that can impose additional checks on the traffic to enforce security policies required by the organisation.

The main idea of GaaS is to move away from the concept of using source IP addresses as an identifier to control application connectivity. Instead, GaaS uses an application account (it does not have to be specific to GaaS, only requires this account to ‘onboard’ to GaaS) as a form of identification, authentication and to check if the party initiating the connection is authorised to access the internet URL.

How GaaS Proxy Solution is put together

The key components with the GaaS proxy appliance are:

1. LDAP authentication manager — For LDAP authentication, using F5 Access Policy Manager (APM).
2. F5 BIG-IP SSL Orchestrator — Inspecting inbound and outbound traffic to enable anti-virus scanning and prevent sending confidential data out of the organisation.
3. ICAP configuration — Used to execute a remote procedure call to external component e.g. for anti-virus via ICAP protocol.
4. Data group — For grouping of ‘whitelist’ and ‘blacklist’ of URLs by application account ID, GaaS uses the component ‘Data group’ from F5 BIG-IP Local Traffic Manager.
5. iRules, TCL scripts — The main logic where it will be and to perform checks on all HTTP traffic using GaaS as the proxy. This component is also from F5 BIG-IP Local Traffic Manager.

All application accounts that accesses the internet via the GaaS will be assigned with its own dedicated “whitelist” and “blacklist”. By default, all application HTTPS requests are set with a “Deny-all” policy. These rules are managed by using ‘Data group’ in the proxy appliance.

What are the differences between “whitelist”, “blacklist” and “Deny-all” policies?

• Whitelist — URLs that has been reviewed and approved for an application account to access.
• Deny-all — Non-approved URLs will be denied by default regardless of application account.
• Blacklist — This will be validated first. URLs that were approved before but became a malicious webpage in future will be blacklisted. The blacklist supersedes the whitelist policy.

An example of blacklist Data group, the censored are the blacklisted URL(s) which will be crafted based on the corporate policy.

Example of a user whitelist base on the application account ‘gaaslvuser’

By using application account as an identifier in conjunction with whitelist and blacklist, the GaaS policies’ management style also solves some of the issues faced by traditional firewall operations.

1. Firewall uses IP addresses as an identifier, thus whenever there is a change or review, the IP addresses are not sufficient to identify which application the change will impact.
2. Complexity in managing firewall rules increases as the organisation grows. This causes the firewall administrators to become more hesitant about removing or changing rules as they have no visibility on which application may be impacted.

The GaaS solves these management issues as firewall administrators can now see which applications will be affected immediately during a policy review/change. Think of attending a graduation ceremony for your child, the receptionist (GaaS in this case) will validate your child name(account ID) against the list of graduates, if your child name is in the list, you will then be allowed into the graduation ceremony venue(URL).

Example of automation workflow to create policy grouping by account ID

GaaS uses API expose by the proxy appliance to create and manage these rules automatically. This indirectly reduces the firewall administrators’ workload and improves the turnaround time for implementing a rule/policy from 3–5 days to just a few minutes in DBS.

GaaS uses the iRules within the proxy appliance to customise different sets of rules for checking whenever it receives a traffic. These sets of rules will decide to ‘Allow’ or ‘Deny’ the traffic.

Example of F5 iRules for policy checking

The iRules within the proxy appliance also helps to capture all traffic logs and efficiently sends them to a log analytic server.

By sending logs to the analytic server, GaaS can use these logs to scan for obsoleted policies and removing them in the process. This removes unnecessary backdoor access due to negligence or complexity in firewall rules changes.

Each of these request needs to be authenticated successfully before the traffic is verified against the access control policy dynamically at the proxy appliance. This way, GaaS does not compromise security and compliance to enable internet access via application hosted in container platform.

Visualization of Access Policy — using F5 BIG-IP Visual Policy Editor

Improved security

GaaS also enabled anti-virus scanning for traffic that is being encrypted with SSL/TLS. Normally, the traffic will not be able to be scanned if it is being encrypted. GaaS solves this by adding an additional layer of workflow that performs SSL inspection to decrypt and encrypt inbound and outbound traffic.

After decrypting the request or response payload, detecting data loss and anti-virus scanning will be possible.

The anti-virus server also updates the signature file daily to keep up with the latest virus trends.

Challenges faced and how to overcome them

Due to anti-virus integration, GaaS also encountered other restrictions for some connection types such as Server Sent Event (SSE) and 2-way-SSL.

What is so special about these 2 connection types that made the anti-virus unable to work?

Server Sent Event

• This is a type of HTTP connection much like web socket, established a persistent connection for streaming information from server to client.
• When a connection is streaming, the anti-virus does not know when to ‘Stop scanning’ or ‘EOF’ to determine if the connection is closed. This cause the anti-virus to continue waiting.

2-way-SSL (Mutual SSL/TLS)

• Some applications have the requirement to trust a specific certificate returns from the server, and the server must trust the certificate sent from the client.

One way would be to create a different access point within the proxy appliance that is defined by a different port number. This new access point will need to handle such connectivity types differently.

Conclusion

By removing the dependencies of source IP addresses in internet access control policies, the GaaS enables container technologies to connect to the internet with ease securely. Furthermore, using application accounts as an identifier also makes internet access control policy management simpler. This gives confidence to the administrator who will have to manage these policies.

(This article is also translated in Traditional Chinese language.

大型K8s叢集網路存取如何更安全？星展打造不一樣閘道即服務翻轉存取控管模式 — Goh Chun Hian — Medium)