Battle Testing Istio 1.0

Istio recently reached its 1.0 release milestone and is considered production ready. This post provides a thorough analysis by our SRE team on Istio 1.0 production status, concluding with our recommendations.

This post is structured as follows:

• Installation & Environment
• Key Findings & Recommendations

Proof of Life — Environment & Installation

Istio is based on microservice architecture and the application network functions are handled by various services:

• Sidecar — proxy intercepting ingress and egress traffic, accepting global configuration provided by Pilot
• Pilot — managing proxies and interacting with Kubernetes
• Mixer — receiving telemetry from proxies, enforces ACL, quotas,
• Citadel — provides authentication and identity for services and users, credentials, RBAC

Istio Architecture

More details are available in official documentation here.

The installation is straightforward. There are several environments supported for installation (e.g. plain Docker), but Kubernetes is the one most suitable for Volterra’s needs.

Supported Installation Methods:

• Pre-rendered YAML: This method is well suited for canned environments (demonstrations, evaluations, POCs, etc.); because parameter tuning is impossible. The YAML file can be templated but has 3700 lines, making it an involved effort.
• Helm: This approach is parametrized and better suited for real world deployments. Serious production deployments would require customization.

Given this was a technology evaluation, the approach taken was YAML.

Deployment of Istio 1.0 using static YAML:

Key Findings & Recommendations

Operation complexity

Istio’s feature set grows with each release and with it brings added requirements for operators. Operating Istio requires deep Kubernetes knowledge combined with an intimate understanding of Istio components and configuration. Latest release (1.0) is using 51 CRDs which means 8 new CRDs since release 0.8.

This complexity should be simplified for operators by implementation of Galley composable, consumer and service configs in future releases. Galley is a new component introduced in 1.0 and it’s the top-level config ingestion, processing and distribution component of Istio. The main goal is to address existing config challenges like diagnostics, reusability across components, misconfiguration, validation, etc. This will enable a better operator experience without deep knowledge of any individual component. Galley is responsible for insulating the rest of the Istio components from the details of obtaining user configuration from the underlying platform. It contains Kubernetes CRD listeners for collecting configuration, a Mesh Config protocol (MCP) protocol server implementation for distributing config, and a validation webhook for pre-ingestion validation by Kubernetes API Server [taken doc].

Galley can be disabled in Istio 1.0 and for AWS EKS it must be disabled together with automatic sidecar injection, because EKS doesn’t support enabling admission controllers such as mutating and validating webhooks. The high level design for Galley is available there. (To open design documents, join to Istio Google group)

Managing Ingress Gateway

Istio Gateway, together with VirtualService, is resource replacing native Kubernetes ingress with v1alpha3 resources defined in Istio. It basically provides more advanced configuration than native Kubernetes ingress. However, these resources provide only definition and actual traffic management is performed by ingress gateway.

The ingress gateway is Istio proxy (Envoy) configured to terminate ingress traffic (from outside of the service mesh), according to Gateway and VirtualService definitions. The default installation starts only single ingressgateway pod and allocates only one public IP (using LoadBalancer service). Following limitations must be taken into account:

You can’t have multiple services listening on same port, e.g. only one 3306 port can be exposed outside of service mesh

VirtualService’s path can’t overlap. This can be a problem if you have multiple teams using Istio and expect them to be independent.

Both limitations can be solved by running multiple ingressgateway pods listening on multiple public IPs (VIPs). However, there is no support for simple management of multiple ingressgateway instances and you need to be careful about using proper selectors.

Config validation and synchronization

You can configure Istio using standard kubectl command or via the native istioctl utility. These two methods deliver the same result — creating CRDs in Kubernetes. Note: only istioctl validates your CRD resources before sending it to Kubernetes API server.

There is second validation performed by Istio’s webhook installed in Kubernetes but it will validate only configuration structure and functional validation isn’t implemented. You can easily hit Ingress Gateway overlap problems described above because configuration is syntactically valid but defines non-functional configuration.

Istio 1.0 added several improvements into istioctl command to get status and health. istioctl proxy-status retrieves last sent and last acknowledged xDS sync between Pilot and Envoy(s) and it provides a status overview.

Another new command is the istioctl authn tls-check, which requests Pilot to check authentication policy and destination rules used by each service in service registry, and validates if TLS settings are compatible between them.

StatefulSet and Headless Service support

Headless services are problematic in service mesh environments because there is no network address associated with this service, and thus instances associated with this service are usually connecting in peer-to-peer manner. It’s a challenge to configure mTLS authentication between these services due to difficult traffic detection and interception.

Kubernetes StatefulSet is often used together with headless services and we have discovered problems in communication between instances of a service. We are running Cassandra cluster in StatefulSet and second replica reported:

INFO  13:44:05 Handshaking version with cassandra-0.cassandra.sre.svc.cluster.local/10.244.4.33
INFO  13:44:05 Cannot handshake version with cassandra-0.cassandra.sre.svc.cluster.local/10.244.4.33

These issues can be solved by moving Cassandra out of service mesh or relaxing mTLS authentication policies. These problems are tracked in upstream issues:

https://github.com/istio/istio/issues/5005 https://github.com/istio/istio/issues/1277 https://github.com/istio/istio/pull/6885

Pod probes

Kubernetes leverages pod probes to check whether pods are running and ready to handle traffic. There are several types of probes but HTTP and TCP are the most commonly used. Probe are successful only when Kubelet is able to reach a pod using an HTTP request or via a TCP connection. This approach is problematic because Kubelet daemon isn’t part of service mesh so it can’t reach pods in case of inter-service auth (mTLS) is enabled.

This problem can be solved by:

• Disabling authentication on ports, which is unacceptable for security reasons. Recommended solution is to dedicate TCP port for service probes and disable authentication for this port.
• Disabling pod probes, which isn’t viable for our deployment.
• Using exec probe to run command inside of pod instead of checking it from Kubelet. However, exec probes aren’t recommended because it isn’t checking full path to service and can lead to hidden problems, for example with CNI.

Multicluster gaps

Istio supports multi-cluster deployment with single control plane since release 0.8. There is very interesting blog post, which provides detailed instructions to run Istio in multi-cluster on release 0.8. Google also presented a demo at the last KubeCon, video recording can be found here. Several security and performance improvements were added in release 1.0 including adding Automatic Sidecar injection on remote cluster. Detail design document is available here.

Unfortunately this feature is still problematic for our own production deployments, for the following reasons:

• Pod network’s between clusters needs to be reachable — this is out of scope of Istio and should be done upfront by VPN configuration. In future there is proposal to implement Zero VPN.
• Pilot, Policy, Telemetry and StatsD Endpoints need to be resolvable in remote cluster — currently remote clusters is pointed to Pod IP of control plane cluster. The problem is that the Pod IP will likely change during its lifetime (pod restart or node crash) and it requires complete redeployment of Istio. There was a PR trying to solve this problem by using extra Istio ingress gateway to reach out Istio Pilot, policy and telemetry.
• Istio configuration and state is stored in “master” cluster so it introduces bottleneck together with potential service disruption caused by failure of master cluster.
• Other issues are tracked in following github issue.

Conclusion

Istio 1.0 is marked as the first production ready version with most of core components being stable. While we can softly agree on stability, there are operational issues that can lead to challenges in production environments.

• Fragile operation in dynamic environment because new changes can affect existing workloads.
• The sum of moving parts must be monitored: config changes, sidecar logs, Kubernetes adaptor in mixer, etc.
• Several Kubernetes features aren’t fully integrated: probes, headless services.

Istio and its execution to date have been a great achievement. Given that, we feel it has some work needed to be operated in production by Kubernetes users with little to no knowledge of the system’s underpinnings. As of this moment, and arguably given its 1.0 status this is expected, most users fall in the category of “power users”. In the short term Managed Istio offerings will likely be needed to be consumed. Stay tuned for future posts related to performance, scaling and policy handling.