Multus Installation on Kubernetes

Multus CNI is a container network interface (CNI) plugin for Kubernetes that enables attaching multiple network interfaces to pods. Normally, when you deploy a pod from Kubernetes, it will have single interface so using Multus you will be able to attach multiple interfaces to pod.

First we are making the deployment of Multus with following file:

nano multus-deployment.yaml
---
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
  name: network-attachment-definitions.k8s.cni.cncf.io
spec:
  group: k8s.cni.cncf.io
  version: v1
  scope: Namespaced
  names:
    plural: network-attachment-definitions
    singular: network-attachment-definition
    kind: NetworkAttachmentDefinition
    shortNames:
    - net-attach-def
  validation:
    openAPIV3Schema:
      properties:
        spec:
          properties:
            config:
                 type: string
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
  name: multus
rules:
  - apiGroups: ["k8s.cni.cncf.io"]
    resources:
      - '*'
    verbs:
      - '*'
  - apiGroups:
      - ""
    resources:
      - pods
      - pods/status
    verbs:
      - get
      - update
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
  name: multus
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: multus
subjects:
- kind: ServiceAccount
  name: multus
  namespace: kube-system
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: multus
  namespace: kube-system
---
kind: ConfigMap
apiVersion: v1
metadata:
  name: multus-cni-config
  namespace: kube-system
  labels:
    tier: node
    app: multus
data:
  cni-conf.json: |
    {
      "name": "multus-cni-network",
      "type": "multus",
      "capabilities": {
        "portMappings": true
      },
      "delegates": [
        {
          "cniVersion": "0.3.1",
          "name": "default-cni-network",
          "plugins": [
            {
               "etcd_ca_cert_file":"/etc/calico/pki/ca",
               "etcd_cert_file":"/etc/calico/pki/crt",
               "etcd_endpoints":"https://10.96.232.136:6666",
               "etcd_key_file":"/etc/calico/pki/key",
               "ipam":{
                  "type":"calico-ipam"
               },
               "kubernetes":{
                  "kubeconfig":"/etc/cni/net.d/calico-kubeconfig"
               },
               "log_level":"info",
               "policy":{
                  "type":"k8s"
               },
               "type":"calico"
            },
            {
               "capabilities":{
                  "portMappings":true
               },
               "snat":true,
               "type":"portmap"
            }
          ]
        }
      ],
      "kubeconfig": "/etc/cni/net.d/multus.d/multus.kubeconfig"
    }
---
apiVersion: extensions/v1beta1
kind: DaemonSet
metadata:
  name: kube-multus-ds-amd64
  namespace: kube-system
  labels:
    tier: node
    app: multus
spec:
  updateStrategy:
    type: RollingUpdate
  template:
    metadata:
      labels:
        tier: node
        app: multus
    spec:
      hostNetwork: true
      nodeSelector:
        beta.kubernetes.io/arch: amd64
      tolerations:
      - operator: Exists
        effect: NoSchedule
      serviceAccountName: multus
      containers:
      - name: kube-multus
        image: nfvpe/multus:stable
        env:
        - name: KUBERNETES_NODE_NAME
          valueFrom:
            fieldRef:
              fieldPath: spec.nodeName
        command:
        - /bin/bash
        - -cex
        - |
          #!/bin/bash
          sed "s|__KUBERNETES_NODE_NAME__|${KUBERNETES_NODE_NAME}|g" /tmp/multus-conf/00-multus.conf.template > /tmp/multus-conf/00-multus.conf
          /entrypoint.sh \
            --multus-conf-file=/tmp/multus-conf/00-multus.conf
        resources:
          requests:
            cpu: "100m"
            memory: "50Mi"
          limits:
            cpu: "100m"
            memory: "50Mi"
        securityContext:
          privileged: true
        volumeMounts:
        - name: cni
          mountPath: /host/etc/cni/net.d
        - name: cnibin
          mountPath: /host/opt/cni/bin
        - name: multus-cfg
          mountPath: /tmp/multus-conf/00-multus.conf.template
          subPath: "cni-conf.json"
      volumes:
        - name: cni
          hostPath:
            path: /etc/cni/net.d
        - name: cnibin
          hostPath:
            path: /opt/cni/bin
        - name: multus-cfg
          configMap:
            name: multus-cni-config
kubectl create -f multus-deployment.yaml

After the deployment completed, you can go and check /etc/cni/net.d/00-multus.conf whether this file is created or not. Also, please check that content of /etc/cni/net.d/10-calico.conflist should be same with config-map of “multus-cni-config” in K8S.

In order to install required plugin to your host, first you need to install GO to your hosts.

wget https://dl.google.com/go/go1.13.3.linux-amd64.tar.gz
sudo tar -xvf go1.13.3.linux-amd64.tar.gz
mv go /usr/local   → if you get “directory not empty”, run the go directory with following commands “cd /usr/local” “rm -R go/” “cd” 
export GOROOT=/usr/local/go
export GOPATH=$HOME/Projects/Proj1
export PATH=$GOPATH/bin:$GOROOT/bin:$PATH
go version → check whether go is installed properly or not

After GO installation is finished, we can now install the plugins to your host:

git clone https://github.com/containernetworking/plugins.git
cd plugins
./build_linux.sh
cp bin/macvlan /opt/cni/bin/
ll /opt/cni/bin

Under this directory “/opt/cni/bin”, you will see the plugins that is installed to your host. If you have high available Kubernetes cluster with multiple nodes, you have install these plugins to all of your nodes.

Now we can create CNI in Kubernetes. In this tutorial, we will just create CNI with “macvlan”. First we have create a network-attachment-definitions:

nano macvlan-conf.yaml
apiVersion: "k8s.cni.cncf.io/v1"
kind: NetworkAttachmentDefinition
metadata:
  name: macvlan-conf
spec:
  config: '{
      "cniVersion": "0.3.0",
      "type": "macvlan",
      "master": "bond0.142",
      "mode": "bridge",
      "ipam": {
        "type": "host-local",
        "subnet": "10.154.142.0/24",
        "rangeStart": "10.154.142.200",
        "rangeEnd": "10.154.142.220",
        "routes": [
          { "dst": "0.0.0.0/0" }
        ],
        "gateway": "10.154.142.1"
      }
    }'

In this file, you have set the network interface of the host, subnet, range and gateway details for your CNI. Then, you can create network-attachment-definitions in K8S with following command:

kubectl create -f macvlan-conf.yaml
kubectl get network-attachment-definitions

Now, we are ready to check if CNI works or not so we will create sample pod and check its interfaces.

apiVersion: v1
kind: Pod
metadata:
  name: samplepod
  annotations:
    k8s.v1.cni.cncf.io/networks: macvlan-conf
spec:
  containers:
  - name: samplepod
    command: ["/bin/ash", "-c", "trap : TERM INT; sleep infinity & wait"]
    image: alpine

Important point here is that you have to write your CNI in annotations part. Here we defined it as “k8s.v1.cni.cncf.io/networks: macvlan-conf”. Now we can create this pod:

kubectl create -f samplepod.yaml

After pod is running, we can check it’s network interfaces with following command:

kubectl exec -it samplepod — ip a

We can see that this pod has two interfaces. One of them is internal IP and other is external IP address.

References

1. https://github.com/k8snetworkplumbingwg/reference-deployment/blob/master/multus-calico/multus-daemonset.yml#L1
2. https://tecadmin.net/install-go-on-ubuntu/
3. https://github.com/containernetworking/plugins