Kiali releases 1.34 to 1.39: Feature update

Hello, community! It has been a little more than 4 months since my last feature update. In all this time, there have been six Kiali releases! Quite a lot! Because of that, this post is one of the largest feature updates I have written and, believe it or not, this will be my last feature update and use the time to focus on some other areas of Kiali. My first update was for Kiali version 0.15.0, yes, before v1.0! I’ve enjoyed writing each post, but after two and half years I’m happy to hand it over to a new Kiali maintainer, Hayk Hovsepyan. Thanks Hayk and I’m eager to read your first feature update!

Alright! Let’s start with the content. As always, let me list the recordings of the live presentations, which happened on schedule:

• Kiali v1.34 update — about 16 minutes long
• Kiali v1.35 update — about 10 minutes long
• Kiali v1.36 update — about 25 minutes long
• Kiali v1.37 update — about 24 minutes long
• Kiali v1.38 update — about 19 minutes long
• Kiali v1.39 update — about 30 minutes long

The live demos are always worth to watch, because usually features are demoed. However, if you prefer to read, let’s start with the written updates!

New traffic drop-down in graph

Next to the Namespaces drop-down selector, there is a new Traffic drop-down. This new option lets you select which protocols to include in the graph.

By selecting (or deselecting) protocols, the graph will include (or exclude) the services that are communicating using your protocol selection.

On each protocol, you can choose the source of data to build the graph from. For example, for the TCP protocol you can choose between Received bytes, Sent bytes or Total bytes. Each of these options relates to a metric emitted by Istio. If you didn’t know already, Kiali generates the graph from Istio metrics. So, this Traffic drop-down lets you to change the graph data source and the generated topology could be slightly different.

See this feature working in the following animation:

Graph fixes related to multi-cluster support

There were two small issues related to multi-cluster support: nodes representing ServiceEntries and Idle workloads were not being associated correctly to their cluster. This was leading to a slightly confusing graph because of nodes not being placed into the right box (or compound) node.

For example, this is the generated graph before the fix (left) and after the fix (right) — notice the different placement of the idle node reviews-v1:

We also learned that some Istio installations that are single cluster are using Kubernetes as the default cluster ID. This was triggering the graph mechanism to show cluster ID on the nodes, resulting in a graph like the following one:

This is noisy, because the Kubernetes indicator on each node label is unnecessary. This was improved and single cluster setups that use the de-facto Kubernetes cluster ID will no longer generate a graph with this redundant information. Moreover, the cluster ID is going to be present on the label only if not already provided by some other grouping box.

For example, in the following image, the graph is showing two namespaces. Notice that non-enclosed nodes fallback to showing the namespace name in parenthesis on their labels, while nodes that are inside a grouping box no longer have specify the namespace name, because the box label already contains this information:

There was also a small bug fix that sometimes was causing the graph to show ServiceEntry nodes on the wrong namespace. This bad behavior should no longer happen.

Enhancements of graph’s find/hide feature

In the graph toolbar, the Find and Hide text boxes are now offering a small drop-down menu with some pre-defined presets. In the following screenshot, you can see the default Find presets:

The default presets are quite basic, but you can configure them in the Kiali CR and replace the default presets with the criteria that best works for your environment.

Another enhancement is that this Find/Hide feature is now capable of filtering by Istio principals, when you are using Istio Peer Authentication (or mTLS). Kiali lets you do principal filtering on edges by using the sourceprincipal or the destprincipal keywords to filter by the identity of the source, or the identity of the destination workload, respectively.

For example, in the following screenshot, by using the expression sourceprincipal*=default, we can see that all services in the travel-portal and travel-agencynamespaces are sharing the same default identity, while services in the bookinfo namespaces aren’t using the default identity.

Throughput on graph edges

There is a new Throughput display option in the graph. By enabling this option, the edge labels will show the throughput of the connection either in kb-per-second or byes-per-second. You can choose between throughput of requests, or responses.

This option can help you at identifying bottlenecks on your services. For example, the following graph reveals that responses generated by the productpage-v1 application are bigger than responses of other services:

Graph badges for Istio traffic features supported by Kiali wizards

In case you don’t know yet, Kiali offers configuration wizards for some traffic management features of Istio. The configuration wizards are available via the Actions menu in the Overview tab of the detail page of a service:

If you configure any traffic management via the Kiali wizards, the graph will add badges to the affected services, indicating which features were configured:

As you can see in the screenshot, the side panel will also offer the same information.

Remember that you can always check the Legend of the graph to learn the meaning of each badge of the graph and other artifacts. The following image of the legend has highlighted the badges that are associated with Istio’s traffic routing features:

Gateway hosts in graph

On the previous screenshots you may notice that there is a new “n hosts” legend in some nodes of the graph. You may have guessed that the graph is now capable of showing hostnames for Istio Ingress Gateways and for VirtualServices. To keep a consistent sizing of nodes, the graph offers this list in an expandable way, by hovering the mouser cursor over the hostnames count:

In the previous animation, you can see that there is a new “globe” badge on the istio-ingressgateway node. This badge, indicates that Kiali detected that node as an Istio Ingres gateway. You may think this is redundant because of the workload name, but you can choose any name for your gateways and the badge gives clarity for those cases.

For usability reasons, the graph shows only 5 hostnames. If there are more, the list is truncated with a proper indication, but you still can see the full list in the side panel:

Hostnames in the detail page of Services

Complementary to the previous enhancement, if a Service is attached to an Istio VirtualService or is associated to a Gateway, its detail page will show the list of hostnames that are being used to route (or filter) traffic to the service:

Kiali will show the most specific list of hostnames:

• If the attached VirtualService has a list of hostnames, Kiali will show this list.
• If the attached VirtualService uses the wildcard “*” in its hosts configuration, Kiali will show the hostnames list of the associated Gateways, if any.

Enhanced Overview tabs in detail pages

A lot of people like Kiali because of the topology graph. We also like the graph. Because of this, the overview tab in details pages was rethought.

In the detail pages, the overview tab had a minigraph. We managed to redistribute the information shown in the overview tab to enlarge the minigraph without giving up on the important information.

This is the current look of the overview of a Workload:

And, for reference, this is how it looked in the past:

When comparing the previous design, you can see that some data went away (for example, the workload type, creation date, resource version, etc.) and some data was moved to tooltips (like the health details). We think there is a good trade off, because the removed data didn’t really help in our observability goal, and data that was moved to tooltips tends to be relevant only if there is an issue.

Traces correlation in workload logs view

In the workloads details page, the Logs viewer now supports fetching and showing related traces. Look at it:

Quite awesome, no? This makes the Log viewer a more powerful tool to correlate traces and the logs of the different containers of a pod.

You can see that lines corresponding to spans have a small ⓘ icon. If you click on this icon, you will be redirected to the Traces tab and the relevant span will be pre-selected to let you inspect its details.

The logs tab also had a small bug: Istio gateway weren’t being recognized as envoy proxies. This leaded to unparsed logs and Kiali not offering the dialog with the explanation of each log line. This was fixed and should now be working as expected.

New layout of metric tabs

In the details pages the Inbound metrics and Outbound metrics tabs had a two column layout. Also, the height of the charts was somewhat large.

This layout dates back to the early beginning of Kiali, when there was no expand option available. So, the charts needed to be big enough to be legible. The expand option was included not too much time later but, despite this, the layout has remained mostly untouched. The problem with the old layout is that the user experience is sub-optimal on screens that have resolutions below 1080p. In some cases, only two charts would fit in the screen.

Starting Kiali v1.36, the layout of the metrics was reworked, trying to be more friendly with screens below the 1080p mark:

There is a related enhancement for custom dashboards. If you are familiar with this feature, they already offered the possibility to configure more than two columns. However, the number of rows to show in the screen was limited to two. So, custom dashboards were improved to also support configuration on the number of rows making it possible to configure layouts like the following one:

Pod failure indicator on the Workload detail page

Your workloads may fail for several reasons. A pod in a CrashLoopBackOff state is a common one and this can be because your app is crashing or the Istio sidecar is not starting. Another common problem is a misconfiguration of the container image, leading to an image pull failure.

A pod can fail for other reasons. In any case, it can be important to know that a pod is not in Running state. Kiali was giving some clue about failing pods in the Health indicator of the workload, however the failing pod didn’t have any failure indicator. Now, you will clearly see if a pod is failing, and also if it is failing because of the sidecar, or because of some other container in the pod:

Envoy-related information moved into a tab in the Workload details page

Maybe you remember that you can see the configuration of Envoy proxies in the Workload details page. This information was shown in a modal dialog available through the Actions menu in the Workload overview (read this older post for reference).

For better accessibility and easiness of use, the modal dialog was removed and Envoy configuration was moved as a main tab in the Workload overview page:

In addition to the already commented new Envoy tab, the previous screenshot is highlighting two other changes:

• The Envoy Metrics tab was pulled as a sub-tab into the new Envoy tab, to have all information related to Envoy proxies under the same context.
• The Destination column in the Listeners tab can have links. If you click in these links, Kiali will move to the Routes tab with a preset filter to list only the routes of interest.

Associated Istio configuration in list pages

Sometimes, it can be convenient to see if certain Istio configurations are associated to an Application, Workload or Service. Kiali already offers this in detail pages, of course! However, you may be thinking that this information should also be available on list pages. OK! Here you go:

As you can see, on list pages the Details column will show associated Istio configurations of each item. This is in addition to any other information that was already available on this column.

Taking advantage of this screenshot, you may probably notice that the columns were reorganized. This reorganization was done in the Applications, Workloads and Services list pages. You should notice a more consistent experience across these three list pages: when possible, columns will be on the same place (or ordering).

Multi-cluster fixes for Istio’s canary upgrade support

Istio can be installed using a revision tag. Although Istio installations with a revision tag may not be fully tied to canary upgrades, Istio documentations mentions revision tags mainly as part of a canary upgrade process.

Kiali reads Istio’s sidecar injector ConfigMap as part of its discovery process. Kiali was using a hard coded name for this ConfigMap, but people using revision tags (or doing canary upgrades) may see some issues on a multi-cluster environment which were reflected in Kiali logs.

Bougyman, a community user, helped to fix this issue.

Allow read-only file system in Kiali pod

Kiali was doing some writes to it’s file system and a failure to update . These writes were because of some front-end files needing to change given certain configurations in the Kiali CR. The problem is that this prevented usage of PodSecurityPolicies to enforce pods to have a read-only root file system.

A read-only root file system is a good policy that follows the minimum privilege principle. So, we removed these writes and replaced them with some in-memory operations to be able to allow a read-only root file system for the Kiali pod.

Hopefully, you can now use your favorite PodSecurityPolicies and Kiali should work properly.

OpenID authentication improvements

When OpenID authentication was terminating with an error, there were some cases that resulted in a technical response from Kiali, like outputting a JSON document. This was particularly true when using OpenID’s authorization code flow.

Error handling was improved to show a more user friendly response. Now, you should always see the error in Kiali’s login page:

Another improvement is that now it is possible to configure a custom CA certificate for communications between Kiali and your identity provider. This is useful if your organization has an internally trusted Certificate Authority and your identity server is using a certificate issued by your internal CA. The ability to configure a custom CA certificate prevents disabling certificate validation. Check the updated documentation to learn how to do this configuration.

Custom dashboards configuration superseded to the Kiali CR or annotations

Kiali lets you create and configure custom dashboards to display in the Application or Workload detail pages, and Kiali offers a few built-in dashboards. Historically, configuration of custom dashboards was done via Kubernetes CRD resources.

Although Kubernetes CRD resources are an interesting option for providing extensibility and configuration for applications, sometimes they are not trivial to maintain. Because of the hard maintenance, starting with Kiali v1.36.0 the CustomDashboard CRD was removed from Kiali. Instead, you configure custom dashboards via de Kiali CR, or via annotations in namespaces or workloads. Read the documentation in the Kiali website to learn more.

This is a breaking change if you are using custom dashboards. I know this advice comes a little late, but upgrade with care if you are still using a version prior to v1.36.

Istio’s canary upgrade assistance

If you are performing a canary upgrade of Istio, Kiali can assist you at upgrading or downgrading your namespaces.

In the Overview page, Kiali can offer the upgrade or downgrade option in the kebab menu of each card:

When you either upgrade or downgrade, Kiali will set the needed istio.io/rev label to the namespace and will also restart the pods.

This feature needs to be configured in the Kiali CR. You need to specify the current (or old) revision of Istio, and the updated (or new) revision, as in:

spec:
  external_services:
    istio_canary_revision:
      current: "1-10-3"
      upgrade: "1-11-0"
  kiali_feature_flags:
    istio_upgrade_action: true

Ability to turn off validations

Kiali can validate the Istio CRDs that are present in your cluster. There is a large list of validations that Kiali can perform and will show warnings if any inconsistency is found. The YAML editor of Istio Configs is the typical place where you see validation details:

Depending on your environment, some validations can be false positives, or the failures are expected because of intentional configurations. To avoid confusion to users and hide these unwanted warnings, it is now possible to disable specific validations via the Kiali CR. For example, this configuration:

spec:
  kiali_feature_flags:
    validations:
      ignore:
      - KIA0101
      - KIA0201

will silence the following messages:

• KIA0101 — Namespace not found for this rule
• KIA0201 — More than one DestinationRules for the same host subset combination

The misconfigurations will no longer be shown in the Kiali UI. However, if you inspect the logs of the Kiali pod, you will see that Kiali is going to emit INFO-level log messages for the failed validations.

Validations around ‘exportTo‘ attribute of Istio CRDs

Some Istio CRDs support setting an exportTo attribute to a list of namespaces where the configuration of the CRD will be applied. This is useful for service owners or mesh administrators to keep a strict control over configurations of the mesh. So, Istio CRDs can be created in a namespace where others users don’t have access, but still set configurations to specific namespaces.

Kiali is now doing validations over the exportTo attribute of several Istio CRDs:

• The exportTo attribute of VirtualServices is taken into consideration to emit the “KIA1106 More than one Virtual Service for same host” message if two VirtualServices introduce a conflict over a hostname.
• The exportTo attribute of DestinationRules is taken into consideration to emit the “KIA1107 Subset not found” when checking references from VirtualServices.
• The namespaces listed in the exportTo attribute of VirtualServices, DestinationRules and ServiceEntries is verified to emit the “KIA0005 No matching namespace found or namespace is not accessible”, if needed.

Gateway validation to HTTP routes of VirtualServices

Gateway validation for VirtualServices has been present for quite some time. Specifically, it is validated that referred Gateways exist. This validation was done only on the main list of gateways in the VirtualService, while the HTTP routes also admit specifying gateways. So, the validation was extended to also check the routes, and now you should also get a “KIA1102 — VirtualService is pointing to a non-existent gateway” for the HTTP routes.

Closing this cycle… but stay in touch!

It was a pleasure to write these posts. I hope you enjoyed reading them as much as I enjoyed writing them. Thanks for reading! I may sporadically write in Kiali’s medium publication, but it will be a different kind of post (i.e. it will no longer be a feature update).

I invite you to drop a comment on the first post of Hayk Hovsepyan.

Remember that Kiali is OpenSource and you can contribute to build a better project! It would be nice if you also contribute with written content in Kiali’s Medium publication!