Secure your service mesh with Istio and keep an eye on it with Kiali

In this article, we will explain how to secure your service mesh using Istio’s Sidecar resource to configure the sidecar proxy and visualize the results with Kiali.

Kiali is a Service mesh observability and configuration tool, it provides answers to the questions: What microservices are part of my Istio service mesh? How are they connected? How are they performing?

You can install Kiali following Getting Started at kiali.io.

It is important to fine-tune the set of services that a workload has access to. It is a good practice to give the least privilege. In that sense, we should grant permissions to each workload to communicate with exactly the services it needs to access. This could also help reducing the attack surface in case of a compromised workload in our mesh.

Bookinfo example

We are going to use Istio’s Bookinfo example to show you how to observe external services.

Bookinfo’s architecture is presented in the following image:

Bookinfo’s architecture.

You can install it by following the instructions found on Istio’s docs.

Once you have it installed and you can drive traffic to productpage service, you can see what we call the “Versioned app graph” by going to Kiali.

Bookinfo’s versioned app graph.

Unwanted requests between services

For example, a developer could contact the ratings service directly instead of using the review service. Let’s simulate this by sending requests from productpage to ratings service.

export PRODUCTPAGE_POD=$(kubectl get pod -l app=productpage -o jsonpath={.items..metadata.name})
kubectl exec -it $PRODUCTPAGE_POD -c productpage -- python -c "import requests;r = requests.get('http://ratings:9080/ratings/0'); print r;print r.json()"

You should get a similar message to let you know that the request succeeded

<Response [200]>
{u'ratings': {u'Reviewer2': 4, u'Reviewer1': 5}, u'id': 0}

After waiting for some seconds, we can refresh the graph and see the request from productpage to ratings.

Bookinfo’s productpage sending requests to ratings service directly.

Now we can see within Kiali that there is a request between these two services. That connection should not exist. As a next step, we are going to show you how to block these requests. We want to avoid this kind of unintentional changes to the architecture.

Preventing unwanted requests between services

There are multiple ways to prevent unwanted requests between services. One way is to use Kubernetes network policies, this, however, requires that you use a network plugin that supports NetworkPolicy. Our other options are Istio’s RBAC resources and Istio’s Sidecar resource.

RBAC resources define roles with rules of what is accessible and bind them to subjects, while Sidecar resources describe the configuration of the sidecar proxy that resides next to your workload. In this article, we are going to use the Sidecar resource to prevent unwanted requests.

First, we need to ensure that we are in the correct mode. To prevent our service mesh to allow unwanted outbound traffic, we need to set the outboundTrafficPolicy mode to REGISTRY_ONLY, by default Istio comes with ALLOW_ANY which would prevent these settings from working.

Let’s start by only allowing productpage send requests to the services reviews and details. For that, we are going to apply the following Sidecar configuration.

Let’s slow down a bit to explain the contents of this file; the workloadSelector is a way to specify which workload is going to be affected by this configuration. In this case, we are using the label app=productpage to identify all the workloads that have this label. We could have used the version=v1 if we wanted to be more specific or any other label that is applied to our target workload.

Note: According to Istio’s documentation, there can be at most one Sidecar resource that selects a given workload, if two or more do, the results will be undefined.

The ingress specifies the inbound traffic listener on the sidecar proxy, we are not using it in this example. The egress allows us to specify the outbound traffic listener on the sidecar proxy, among other things it lets us define the service hosts (using FQDN format). We have defined bookinfo/reviews and bookinfo/details to ensure we can only send requests to these two services.

Notice that we also include “istio-system/*”. If we didn’t, we would lose all the metrics, this is because the Sidecar would not allow requests to Istio’s mixer. At the time of writing, it is being discussed if the Sidecar should be configured automatically to talk to the required Istio’s services or if a warning is enough. However, this requirement is temporary and will be removed in the future.

Apply the configuration:

kubectl apply -f https://git.io/fjXSQ

Now that it has been applied, lets try again the previous request.

export PRODUCTPAGE_POD=$(kubectl get pod -l app=productpage -o jsonpath={.items..metadata.name})
kubectl exec -it $PRODUCTPAGE_POD -c productpage -- python -c "import requests;r = requests.get('http://ratings:9080/ratings/0'); print r;print r.json()"

You should get a noisy error message like the following:

<Response [502]>
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/usr/local/lib/python2.7/site-packages/requests/models.py", line 897, in json
    return complexjson.loads(self.text, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/simplejson/__init__.py", line 518, in loads
    return _default_decoder.decode(s)
  File "/usr/local/lib/python2.7/site-packages/simplejson/decoder.py", line 370, in decode
    obj, end = self.raw_decode(s)
  File "/usr/local/lib/python2.7/site-packages/simplejson/decoder.py", line 400, in raw_decode
    return self.scan_once(s, idx=_w(s, idx).end())
  File "/usr/local/lib/python2.7/site-packages/simplejson/scanner.py", line 79, in scan_once
    return _scan_once(string, idx)
  File "/usr/local/lib/python2.7/site-packages/simplejson/scanner.py", line 39, in _scan_once
    raise JSONDecodeError(errmsg, string, idx)
simplejson.errors.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
command terminated with exit code 1

This means that our Sidecar configuration got applied correctly, we are now getting an HTTP error when sending requests to ratings service from productpage.

Bookinfo’s productpage no longer able to send requests to ratings service.

If you go to Kiali you can see the error requests from productpage to an “unknown” service. This is because the service is no longer known to productpage’s Sidecar.

We could add similar Sidecar resources for the different workloads we have in our mesh, to ensure that only valid requests are made.

Cleaning up

Remove Istio resources that were created

kubectl delete sidecar sidecar-productpage

Conclusion

A Sidecar resource can be used to fine-tune how the sidecar proxy behaves. In this article we showed how it can be used to prevent our services in the mesh from communicating to the wrong services and breaking our mesh architecture. Further work can be done with Istio’s RBAC to ensure we e.g. only allow requests to ratings service with a GET request on the /ratings path from within a review service.

Kiali can be used to observe your service mesh to detect errors in it.

Kiali always welcomes contributions in every form. This can be code, but also bug reports, feature requests, translations and much more. You can follow us on twitter, see our videos at youtube, talk with us on IRC or drop us a message in our users and developers group.