How to restrict Kubernetes ingress resources via Open Policy Agent?

Today I will explain how we can restrict ingress resources through Open Policy Agent and also we will deploy Open Policy Agent.

By default, a Kubernetes user who is authorized to create an ingress resource can create an ingress resource with “any” hostname.

For example, let’s look at the example ingress resource below.

We see that it was created with a hostname that we do not want to be used or that we do not own. We can prevent this and similar situations using Open Policy Agent.

But first of all, what is Open Policy Agent?

Open Policy Agent is an open-source, general-purpose policy engine that decouples policy decisions from other application responsibilities.

Prerequisites:

• Kubernetes 1.20+ Cluster
• A server with Docker (For OPA Agent)
• Openssl (For creating TLS)

Open Policy Agent Installation:

First, let’s create a namespace for the OPA.

kubectl create namespace opa

Now we need to create TLS credentials. We have to use TLS for communication between Kubernetes and OPA.

openssl genrsa -out ca.key 2048
openssl req -x509 -new -nodes -sha256 -key ca.key -days 100000 -out ca.crt -subj "/CN=admission_ca"

cat >server.conf <<EOF
[ req ]
prompt = no
req_extensions = v3_ext
distinguished_name = dn

[ dn ]
CN = opa.opa.svc

[ v3_ext ]
basicConstraints = CA:FALSE
keyUsage = nonRepudiation, digitalSignature, keyEncipherment
extendedKeyUsage = clientAuth, serverAuth
subjectAltName = DNS:opa.opa.svc,DNS:opa.opa.svc.cluster,DNS:opa.opa.svc.cluster.local
EOF

openssl genrsa -out server.key 2048
openssl req -new -key server.key -sha256 -out server.csr -extensions v3_ext -config server.conf
openssl x509 -req -in server.csr -sha256 -CA ca.crt -CAkey ca.key -CAcreateserial -out server.crt -days 100000 -extensions v3_ext -extfile server.conf

Now let’s create a secret to store TLS credentials.

kubectl create secret tls opa-server --cert=server.crt --key=server.key --namespace opa

Our next step is to create two policies.
The first policy will only allow the creation of an ingress resource using the hostnames we allow.

The second one will prevent the same hostname from being used between different namespaces.

Create directory for policies.

mkdir policies && cd policies

Let’s define the first policy.

cat >ingress-allowlist.rego <<EOF
package kubernetes.admission

import data.kubernetes.namespaces

operations := {"CREATE", "UPDATE"}

deny[msg] {
 input.request.kind.kind == "Ingress"
 operations[input.request.operation]
 host := input.request.object.spec.rules[_].host
 not fqdn_matches_any(host, valid_ingress_hosts)
 msg := sprintf("invalid ingress host %q", [host])
}

valid_ingress_hosts := {host |
 allowlist := namespaces[input.request.namespace].metadata.annotations["ingress-allowlist"]
 hosts := split(allowlist, ",")
 host := hosts[_]
}

fqdn_matches_any(str, patterns) {
 fqdn_matches(str, patterns[_])
}

fqdn_matches(str, pattern) {
 pattern_parts := split(pattern, ".")
 pattern_parts[0] == "*"
 suffix := trim(pattern, "*.")
 endswith(str, suffix)
}

fqdn_matches(str, pattern) {
    not contains(pattern, "*")
    str == pattern
}
EOF

and the second policy

cat >ingress-conflicts.rego <<EOF
package kubernetes.admission

import data.kubernetes.ingresses

deny[msg] {
    some other_ns, other_ingress
    input.request.kind.kind == "Ingress"
    input.request.operation == "CREATE"
    host := input.request.object.spec.rules[_].host
    ingress := ingresses[other_ns][other_ingress]
    other_ns != input.request.namespace
    ingress.spec.rules[_].host == host
    msg := sprintf("invalid ingress host %q (conflicts with %v/%v)", [host, other_ns, other_ingress])
}
EOF

Finally, let’s define a main policy that imports the restrict hostname and conflict policies and provides an overall policy decision.

cat >main.rego <<EOF
package system

import data.kubernetes.admission

main := {
  "apiVersion": "admission.k8s.io/v1",
  "kind": "AdmissionReview",
  "response": response,
}

default uid := ""

uid := input.request.uid

response := {
    "allowed": false,
    "uid": uid,
    "status": {
        "message": reason,
    },
} {
    reason = concat(", ", admission.deny)
    reason != ""
}

else := {"allowed": true, "uid": uid}
EOF

Now we need to create an OPA bundle and publish it.

cat > .manifest <<EOF
{
    "roots": ["kubernetes/admission", "system"]
}
EOF

curl -L -o opa https://openpolicyagent.org/downloads/latest/opa_linux_amd64
chmod 755 opa
./opa build -b .

Now we will publish the OPA Bundle on an nginx server.
Nginx server must be accessible by Kubernetes cluster.

docker run --rm --name bundle-server -d -p 8888:80 -v ${PWD}:/usr/share/nginx/html:ro nginx:latest

Run the following command by replacing the IP Address field on line 90 with the IP address of the server where nginx server is running.

cat > admission-controller.yaml <<EOF
# Grant OPA/kube-mgmt read-only access to resources. This lets kube-mgmt
# replicate resources into OPA so they can be used in policies.
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: opa-viewer
roleRef:
  kind: ClusterRole
  name: view
  apiGroup: rbac.authorization.k8s.io
subjects:
- kind: Group
  name: system:serviceaccounts:opa
  apiGroup: rbac.authorization.k8s.io
---
# Define role for OPA/kube-mgmt to update configmaps with policy status.
kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  namespace: opa
  name: configmap-modifier
rules:
- apiGroups: [""]
  resources: ["configmaps"]
  verbs: ["update", "patch"]
---
# Grant OPA/kube-mgmt role defined above.
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  namespace: opa
  name: opa-configmap-modifier
roleRef:
  kind: Role
  name: configmap-modifier
  apiGroup: rbac.authorization.k8s.io
subjects:
- kind: Group
  name: system:serviceaccounts:opa
  apiGroup: rbac.authorization.k8s.io
---
kind: Service
apiVersion: v1
metadata:
  name: opa
  namespace: opa
spec:
  selector:
    app: opa
  ports:
  - name: https
    protocol: TCP
    port: 443
    targetPort: 8443
---
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: opa
  namespace: opa
  name: opa
spec:
  replicas: 1
  selector:
    matchLabels:
      app: opa
  template:
    metadata:
      labels:
        app: opa
      name: opa
    spec:
      containers:
        # WARNING: OPA is NOT running with an authorization policy configured. This
        # means that clients can read and write policies in OPA. If you are
        # deploying OPA in an insecure environment, be sure to configure
        # authentication and authorization on the daemon. See the Security page for
        # details: https://www.openpolicyagent.org/docs/security.html.
        - name: opa
          image: openpolicyagent/opa:0.50.1
          args:
            - "run"
            - "--server"
            - "--tls-cert-file=/certs/tls.crt"
            - "--tls-private-key-file=/certs/tls.key"
            - "--addr=0.0.0.0:8443"
            - "--addr=http://127.0.0.1:8181"
            - "--set=services.default.url=http://YOURIP:8888"
            - "--set=bundles.default.resource=bundle.tar.gz"
            - "--log-format=json-pretty"
            - "--set=status.console=true"
            - "--set=decision_logs.console=true"
          volumeMounts:
            - readOnly: true
              mountPath: /certs
              name: opa-server
          readinessProbe:
            httpGet:
              path: /health?plugins&bundle
              scheme: HTTPS
              port: 8443
            initialDelaySeconds: 3
            periodSeconds: 5
          livenessProbe:
            httpGet:
              path: /health
              scheme: HTTPS
              port: 8443
            initialDelaySeconds: 3
            periodSeconds: 5
        - name: kube-mgmt
          image: openpolicyagent/kube-mgmt:2.0.1
          args:
            - "--replicate-cluster=v1/namespaces"
            - "--replicate=networking.k8s.io/v1/ingresses"
      volumes:
        - name: opa-server
          secret:
            secretName: opa-server

EOF

And apply the configuration.

kubectl apply -f admission-controller.yaml

Now let’s create the manifest file that will register the OPA as an admission controller. This webhook will ignore namespaces labeled with “openpolicyagent.org/webhook=ignore”.

cat > webhook-configuration.yaml <<EOF
kind: ValidatingWebhookConfiguration
apiVersion: admissionregistration.k8s.io/v1
metadata:
  name: opa-validating-webhook
webhooks:
  - name: validating-webhook.openpolicyagent.org
    namespaceSelector:
      matchExpressions:
      - key: openpolicyagent.org/webhook
        operator: NotIn
        values:
        - ignore
    rules:
      - operations: ["CREATE", "UPDATE"]
        apiGroups: ["*"]
        apiVersions: ["*"]
        resources: ["*"]
    clientConfig:
      caBundle: $(cat ca.crt | base64 | tr -d '\n')
      service:
        namespace: opa
        name: opa
    admissionReviewVersions: ["v1"]
    sideEffects: None
EOF

Before applying webhook we need to label kube-system and opa namespaces for ignoring.

kubectl label ns kube-system openpolicyagent.org/webhook=ignore
kubectl label ns opa openpolicyagent.org/webhook=ignore

Now we can apply the webhook yaml.

kubectl apply -f webhook-configuration.yaml

And after that we can show the webhook requests.

kubectl logs -l app=opa -c opa -f -n opa

Time to try our policies are working correctly or not.

Let’s create a namespace and in this namespace, ingress resources can only be created with the suleyman.lab hostname.

cat > web-namespace.yaml <<EOF
apiVersion: v1
kind: Namespace
metadata:
  annotations:
    ingress-allowlist: "*.suleyman.lab"
  name: web
EOF

kubectl apply -f web-namespace.yaml

Let’s test it with a valid ingress source.

cat > valid-ingress.yaml <<EOF
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: valid-ingress
spec:
  rules:
  - host: dev.suleyman.lab
    http:
      paths:
      - pathType: ImplementationSpecific
        path: /
        backend:
          service:
            name: nginx
            port:
              number: 80
EOF

kubectl apply -f valid-ingress.yaml -n web

We created ingress resource successfully!

Now let’s create an ingress resource with a hostname that we do not allow.

Request denied! Because according to our OPA Policy, only ‘*.suleyman.lab’ can be used in this namespace.

I hope you enjoyed reading and trying it out.
See you in the next articl

You can find my other articles by clicking here.

LinkedIn

Resources:

https://www.openpolicyagent.org/docs/latest/