Deploy Istio enabled application on IBM Cloud Private in a non-default namespace

Overview

This article reviews the problems you might encounter while deploying an Istio enabled application in a non-default namespace in an IBM Cloud Private environment.

I will also cover how to troubleshoot and fix these issues.

Ingredients

• IBM Cloud Private 2.1.0.1 or later
• A user with cluster administrator role
• Configured kubectl for the administrator role. To install and setup kubectl cli, see here .
• Install Istio on IBM Cloud Private. To download and install Istio, see here.

This article uses the Bookinfo application as a sample application. You can find more information about the Bookinfo application here.

Step-by-step

1. Create a new namespace

Create a new namespace called bookinfo. This namespace is used to deploy the Bookinfo application. To create the namespace, run following command.

kubectl create namespace bookinfo

2. Deploy the Istio enabled application in the new namespace

From the Istio folder, run the following command to manually inject the Istio sidecar into the Bookinfo application. This command also deploys the Istio sidecar in the bookinfo namespace.

kubectl apply -n bookinfo -f <(istioctl kube-inject --debug -f samples/bookinfo/kube/bookinfo.yaml)

3. Verify the deployment

Run the following command to verify the deployment.

kubectl get deployments -n bookinfo

Your output should resemble the following:

yus-macbook-pro:istio-0.6.0 ycao$ kubectl get deployments -n bookinfo
NAME           DESIRED CURRENT UP-TO-DATE AVAILABLE AGE
details-v1     1       0       0          0         2m
productpage-v1 1       0       0          0         2m
ratings-v1     1       0       0          0         2m
reviews-v1     1       0       0          0         2m
reviews-v2     1       0       0          0         2m
reviews-v3     1       0       0          0         2m

If you are wondering why none of your deployments are working, continue to the next section where I show you how to troubleshoot.

4. Troubleshoot and identify the problem

Let’s run some commands to see what the problem is.

1. Review the deployment details.

kubectl get deployments details-v1 -n bookinfo -o yaml

i. In the status section I found the following information.

message: 'pods "details-v1-7986ddbd99-" is forbidden: unable to validate against
   any pod security policy: [spec.initContainers[0].securityContext.privileged:
   Invalid value: true: Privileged containers are not allowed capabilities.add:
   Invalid value: "NET_ADMIN": capability may not be added spec.initContainers[1].securityContext.privileged:
   Invalid value: true: Privileged containers are not allowed spec.containers[1].securityContext.privileged:
   Invalid value: true: Privileged containers are not allowed]'
 reason: FailedCreate
 status: "True"
 type: ReplicaFailure

OK. That’s why the deployment was not working. There are two main problems here:

• spec.initContainers[0].securityContext.privileged: Invalid value: true: Privileged containers are not allowed
• capabilities.add: Invalid value: “NET_ADMIN”: capability may not be added

Let’s take a look at the output again.

ii. In the initContainers section I found following information.

initContainers:
  - args:
    - -p
    - "15001"
    - -u
    - "1337"
    image: docker.io/istio/proxy_init:0.6.0
    imagePullPolicy: IfNotPresent
    name: istio-init
    resources: {}
    securityContext:
      capabilities:
        add:
        - NET_ADMIN
      privileged: true

OK, now it’s confirmed. The sidecar that Istio injected into the Bookinfo application requires the permission to a) run privileged containers and b) add Linux capability — referred to as NET_ADMIN.

5. Fix the problem

Now that we have identified the problem and figured out the root cause, we can try to fix it.

There are a couple of way to fix this.

Fix 1

The easiest way is by using the pre-configured clusterrole in IBM Cloud Private. IBM Cloud Private ships with a clusterrole called privileged. This clusterrole is preconfigured to use a pod security policy called privileged. You can get more details by running the following commands:

kubectl get clusterrole privileged -o yaml
kubectl get psp privileged -o yaml

By looking at the yaml definition of privileged pod security policy, you can see that it already includes the permission required to run the Istio sidecar.

apiVersion: extensions/v1beta1
kind: PodSecurityPolicy
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"extensions/v1beta1","kind":"PodSecurityPolicy","metadata":{"annotations":{},"name":"privileged","namespace":""},"spec":{"allowedCapabilities":["*"],"fsGroup":{"rule":"RunAsAny"},"hostIPC":true,"hostNetwork":true,"hostPID":true,"hostPorts":[{"max":65535,"min":1}],"privileged":true,"readOnlyRootFilesystem":false,"runAsUser":{"rule":"RunAsAny"},"seLinux":{"rule":"RunAsAny"},"supplementalGroups":{"rule":"RunAsAny"},"volumes":["*"]}}
  creationTimestamp: 2018-03-19T20:09:04Z
  name: privileged
  resourceVersion: "149"
  selfLink: /apis/extensions/v1beta1/podsecuritypolicies/privileged
  uid: 5fa44795-2bb1-11e8-ab68-005056a005fd
spec:
  allowPrivilegeEscalation: true
  allowedCapabilities:
  - '*'
  fsGroup:
    rule: RunAsAny
  hostIPC: true
  hostNetwork: true
  hostPID: true
  hostPorts:
  - max: 65535
    min: 1
 privileged: true
 runAsUser:
   rule: RunAsAny
 seLinux:
   rule: RunAsAny
 supplementalGroups:
   rule: RunAsAny
 volumes:
 - '*'

So what we need to do now is to apply this pod security policy to the bookinfo namespace by creating the clusterrolebinding.

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: istio-user
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: privileged
subjects:
- kind: ServiceAccount
  name: default
  namespace: bookinfo

To apply this fix:

Copy the above code snippet and save it as psp.yaml. Then run following command to create the clusterrolebinding.

kubectl apply -f psp.yaml

As soon as IBM Cloud Private detects the change, the Bookinfo application should start to deploy. You can verify this by checking the deployments again.

kubectl get deployments -n bookinfo

This time, the output should resemble the following:

NAME           DESIRED CURRENT UP-TO-DATE AVAILABLE AGE
details-v1     1       1       1          1         25m
productpage-v1 1       1       1          1         25m
ratings-v1     1       1       1          1         25m
reviews-v1     1       1       1          1         25m
reviews-v2     1       1       1          1         25m
reviews-v3     1       1       1          1         25m

Fix 2

If you don’t want to reuse the preconfigured pod security policy above, since it includes extra permissions that the Istio sidecar does not need. Below is a dedicated policy that I created to deploy and run an Istio enabled application.

apiVersion: extensions/v1beta1
kind: PodSecurityPolicy
metadata:
  name: istio
spec:
  privileged: true
  allowedCapabilities:
  - 'NET_ADMIN'
  fsGroup:
    rule: RunAsAny
  runAsUser:
    rule: RunAsAny
  seLinux:
    rule: RunAsAny
  supplementalGroups:
    rule: RunAsAny
  volumes:
  - '*'
 
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: istio
rules:
- apiGroups:
  - extensions
  resourceNames:
  - istio
  resources:
  - podsecuritypolicies
  verbs:
  - use

---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: istio-user
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: istio
subjects:
- kind: ServiceAccount
  name: default
  namespace: bookinfo

To apply this fix:

Copy the above code snippet and save it as psp-all-in-one.yaml. Run the following command to create the clusterrolebinding.

kubectl apply -f psp-all-in-one.yaml

Conclusion

Istio needs to inject sidecar into your original application. However, the sidecar requires extra permissions to a) run privileged container and b) add Linux capability — NET_ADMIN. So when you create a new namespace on IBM Cloud Private, you also need to apply corresponding pod security policies to the new namespace to allow Istio sidecar to work.

In this article, I showed you a step by step example on how to deploy Istio enabled application to a non-default namespace on IBM Cloud Private, how to troubleshoot and how to fix the problem. Finally, I also provided a sample pod security policy with minimum permissions required to deploy an Istio enabled application.

Originally published at developer.ibm.com.