Visualizing Istio external traffic with Kiali
In this article we will show you how to observe traffic to external services using Kiali.
Kiali is a Service mesh observability and configuration tool, it provides answers to the questions: What microservices are part of my Istio service mesh? How are they connected? How are they performing?
You can install Kiali following Getting Started at kiali.io.
Observe external services
Suppose that you have an application using several third party services to store files, send messages, write tweets, etc. It is useful to know how much traffic is going off your mesh to these services, for example, you might want to know how many requests are directed to twitter or how much data is being sent to Dropbox. Also knowing if these requests are successful or if they fail.
Istio provides a resource called Service Entry. It allows to plug additional services into your mesh so that other services can access these manually defined resources. These resources can be part of your mesh (e.g. VMs) or services external to the mesh (e.g. Twitter, Dropbox, etc).
Bookinfo example
We are going to use Istio’s Bookinfo example to show you how to observe external services. You can install it by following the instructions found on Istio’s docs.
Once you have it installed and you are able to drive traffic to productpage service, you can see what we call the “Versioned app graph” by going to Kiali.
Configuring external traffic
Using Bookinfo example, there is a service that could optionally request the book information using Google API, to enable it run the following command:
kubectl set env deployment/details-v1 ENABLE_EXTERNAL_BOOK_SERVICE=true
Now the service requests the book details from Google API instead of using a fixed set of values. Lets configure the ServiceEntry to be able to see it in Kiali.
kubectl apply -f https://gist.github.com/josejulio/68ccb1162231af3e9f298f746b5f5b96/raw/bc5090305f068ad95bea2d81a1121a04a32682a6/googleapi-se.yaml
Now that the ServiceEntry has been created, we can test it by sending requests to Bookinfo and check Kiali’s graph page again.
Now we can see the requ… Wait a minute. You might be wondering why the edge looks different than the other edges. It is not because is an external service, in fact that kind of edge means that we are looking at TCP traffic, as opposed to HTTP.
Why are we looking at TCP if it is sending HTTPS requests?
I had the same question and it turns out that Istio can’t recognize it as an HTTP request, because the request is encrypted (HTTPS), so Istio (Envoy in this case) can only see a bunch of TCP traffic.
But I really want to see the requests, what can I do?
Further configuration to external traffic
Luckily, there is a way to do that. Instead of sending HTTPS requests from the pod, use HTTP and let Istio upgrade the request to HTTPS. In that way, Istio can get the headers of the requests (and do its magic) while ensuring that the traffic gets out of the mesh encrypted.
Lets do that by applying the following yaml file.
kubectl apply -f https://gist.github.com/josejulio/68cfd211ec32ecf20b77534576c8bd52/raw/f66c0e546a1ecaadcf2323459a7581e9efde10d3/googleapi-combo.yaml
Now we need to tell the “details” pod to send the requests without encrypting:
kubectl set env deployment/details-v1 DO_NOT_ENCRYPT=true
Lets inject some more traffic to Bookinfo’s example and then go back to the graph and see that there is HTTP traffic from “details” to Google API.
Some notes, you could end up in the situation where you see two edges going from “details” to the ServiceEntry.
The gray edge means that Kiali no longer detects any traffic. In this case, it is a separate edge because that edge represents the old TCP traffic, after some minutes it will stop appearing in the graph.
Something else that could happen is that the edge from details to the ServiceEntry turns red. A red edge means that we have some error in our traffic. In this case, it is because we are accessing Google API anonymously and Google has a low user rate limit. In my tests, injecting traffic to the mesh every 20 seconds does not bring any error. Inject traffic every 2 seconds and you get something similar:
Cleaning up
Remove Istio resources that were created.
kubectl delete serviceentry www.googleapis.com
kubectl delete virtualservice www.googleapis.com
kubectl delete destinationrule www.googleapis.com
Remove environments that were added to the “details” pod.
kubectl set env deployment/details-v1 ENABLE_EXTERNAL_BOOK_SERVICE- DO_NOT_ENCRYPT-
Conclusion
In this article we showed how to observe requests external to Istio using Kiali. But be aware that with this approach the traffic inside the mesh is not encrypted, an attacker that is able to penetrate the node of the application would still be able to see the unencrypted request.
Note: As the time of writing, there is a bug that affects external requests over HTTPS that don’t have a matching ServiceEntry. These requests are considered in the traffic of any other ServiceEntry that has HTTPS. So if you see unknown TCP traffic, is possible that said traffic is being incorrectly counted towards that ServiceEntry.
