Troubleshooting Envoy with Kiali
Inspect and debug your Envoy configuration
A brief introduction
Before we start diving into Envoy and its configuration, let’s try to answer the following questions:
What is Envoy?
Envoy is a high performance proxy design for service oriented applications.
Why is it needed?
Envoy provides built-in functionality to cope with challenges around micro services architectures, some of these are:
- Retries, delays, circuit breakers and fault injection
- Dynamic service discovery and load balancing
- Traffic management and routing for HTTP, gRPC, WebSocket and TCP traffic
- Security policies and rate limiting
The powerful aspect of a proxy that has the above functionality is that developers are not required to update their applications, the application’s functionality is augmented by Envoy.
For more features that are added by Envoy take a look at Envoy architecture.
How is it used by Istio?
Istio uses Envoy to mediate all inbound and outbound traffic for all services in the service mesh.
Typically on Kubernetes it is deployed alongside each application container (sidecar container pattern) forming a data plane. Virtual machines can be part of the mesh too, where Envoy is installed directly on them.
For more information, take a look at Istio architecture.
Where is its configuration located?
There is not a single Envoy configuration but instead several configurations, a configuration per Envoy instance.
Istio configures every Envoy according to the mesh configuration (thanks to VirtualServices, DestinationRules among others resources).
Why do we need to care about theses configurations?
The majority of the time, Istio will handle the configuration properly with no need for inspection or debugging, but there will be times where for any reason, a bug, or a misunderstood configuration, the Envoy instances will get configurations that do not reflect what we are trying to do. For these cases, taking a look directly into Envoy configuration will help to detect problems.
How can we inspect and debug these configurations?
The following documentation is a good place to start:
Another great way is with Kiali, and we will see it in action in the following sections.
Kiali and the Envoy logs
Envoy generates logs that can help explain why a configuration is not working, or to expose errors occurring at a specific time.
The logs are available in the Logs section in the Workloads page as it is shown next:
Notice that we can select to see the application logs, the Envoy logs or both (time sorted, with different colors).
Another great tool to inspect Envoy logs is with the Envoy Access Log details view. Every Envoy log line has a little indicator on the left, clicking the indicator will show the log line processed as follows:
Kiali and the Envoy configuration
Typically, Envoy configuration is very large, so finding information and debugging tends to be hard and time consuming. For that reason, Kiali adds a way to see the configuration in a different and well organized manner, including links to related resources like VirtualServices and DestinationRules among others.
To better understand what follows, look at Envoy terminology.
On every workload (which has an Envoy injected), the information can be seen in the Envoy tab as follows:
This page shows the configuration split into several resources like clusters, listeners and routes, also the ability to view the bootstrap configuration, and even the full configuration:
As a reminder, a workload is typically a set of related pods and each of them has an Envoy injected. For that reason, we can select which specific pod to inspect using a pod selector.
Lastly, the metrics related to the Envoy being inspected can be seen too:
A walk through Envoy configuration
For this demonstration, the following example based on bookinfo application will be used:
The main goal of the demo is to route requests based on user identity, every request that belongs to Jason, will be routed to reviews-v2.
Our goal is to inspect how that configuration is translated into Envoy configuration.
We will configure the routing rule for the required scenario using Kiali wizards:
Requests will start from the productpage application to the reviews application at port 9080. From productpage application perspective, there will be a listener for that port and we can find it on the Envoy tab filtering by the 9080 port:
We can see the listener and also that it is related to a route, we can easily inspect the route using the link in the listener that takes us straight to the routes with the correct filters applied:
Also notice that some routes have a related VirtualService, created by the wizard. The link will take us to the VirtualService overview, where we can check the created configuration:
In the clusters, if we filter by the 9080 port, we will see all possible clusters and also links to related DestinationRules:
Finally, we can look directly into the full configuration, for example searching for the name Jason, to see the configuration we’ve recently created:
Final words
This functionality aims to give more visibility to Envoy information, facilitate debugging and also to lay the groundwork to continue improving how Envoy related information is presented.
Feedback is appreciated and welcomed, you can reach us on Slack and Libera Chat. Follow us on Twitter to stay in touch for latest news. Thanks for reading!.
