Openshift Egress Traffic
I wrote this brief article to explain how actually the traffic leaves an openshift cluster to reach external endpoints because I have read many articles explaining the ingress traffic within openshift clusters but very few about the opposite case “How the internal openshift components reach the outside services?”. In the following paragraphs, I will try to explain how egress traffic leaves openshift in a simple way as far as I can.
Egress Traffic within Openshift
So what are the sources of egress traffic in the openshift cluster? Or in other words, what are the openshift components that try to communicate with services outside the cluster network?
The openshift cluster originates two types of egress traffic:
1. Platform Traffic.
2. Applications Traffic.
Platform traffic originates from openshift resources and it can be something like DNS resolution, connection to a proxy server, NTP server, external monitoring system, or integration with an active directory. This type of traffic only leaves using the node’s IP “the IP of VM itself” on which the resources run on.
Application traffic originates when application pods in openshift want to access external services like an external database or any endpoint over the internet. From now on we will discuss the possible approaches of how can applications within openshift access external services.
How The Traffic leaves the cluster?
Node IP
The default behavior in openshift is that all traffic either from the platform or from the applications is going to originate from the node’s IP.
Here we have two projects; project A and Project B. Project B has only one pod running in node 1, and Project A has two pods running in both node 1 and node 2. Also, we have an external service which could be a database service in a different zone or any kind of application outside the openshift cluster network. Besides, there is a firewall between the cluster network and the external service network. In this approach to allow the application in project A to communicate with external service we have to allow traffic comes from the node’s IP on which the application run; in this case node 1 and node 2 because project A has two replicas in both nodes but here firewall doesn't allow the traffic from node 2, therefore, only the pod in node 1 can reach the external service. Because the firewall allows all traffic that comes from node 1 the application in project B exposes the external service, and we don’t that to happen from a security perspective.
This approach has many cons compared to pos; the single pos I can mention is “it’s simple for a lab environment with nearly no restrictions to care about”. To overcome the node IP model cons there are two other approaches are used in production environments and we will discuss them in the following sections.
Egress Router
In the previous approach, we see how it allows all the applications to expose the external service whether it should or shouldn't connect to the external service; to overcome this issue other egress models are introduced: Egress Router and Egress IP and I am going to start with Egress Router.
Egress Router is another application deployed in the project that we want its applications to connect to the external service.
In the above diagram like the previous scenario, we have a firewall between the cluster network and the external service and we have two projects A and B. The new in this approach is that in project A in which we want the apps the expose the external service we have an egress router that will take the egress traffic from the application and forward it to the external service. But before the pod traffic is sent to the external service through the egress router a must egress service is located between the egress router and the application pod to forward the traffic from the pods to the egress router.
Firstly, In Project A the replica which tries to connect directly to the external service fails because it’s not allowed through the firewall. The other replica communicates with the egress service that later on forward the traffic to the egress router which has an IP in the firewall whitelist of allowed IPs.
Secondly, In project B the application tries to reach the egress service but because of network policies, it’s not allowed to communicate with any other resources outside its namespace and of course, if it tries to connect directly to the external service it will fail because of the firewall.
Note:
It is noteworthy to mention that the default network policy in these examples is denay all the traffic which means that pods can communicate with resources within their namespace but not resources in other namespaces.
Egress IP
The last approach of the three and simpler than the egress router. Egress IP is configured at the namespace level so we can originate the egress traffic of applications from them. Each node on which the application [Pods] runs can be assigned one or more egress IPs which are allocated to the application namespace within the cluster’s node network free IPs, which means the nodes may have multiple IPs; the node IP itself and egress IPs.
In the above scenario application in Project B tries to connect to the external service but as we have mentioned before, the firewall will block the traffic coming from the IPs not in its whitelist. On the contrary, the application of Project A can reach the external service because pods in Project A forward their traffic through egress IPs that are assigned to their namespace, and these IPs are allowed through the firewall.
Conclusion
In conclusion, after we have discussed the egress approaches in the openshift cluster; we can tell that there is no best approach each one of them has its own suitable environment. For example, node IP is suitable for lab env for its simplicity, and egress IPs for their high availability.
I hope this article gives you a teeny tiny glimpse of how egress in openshift works.
Here are worth mentioning references: