Boost Istio Service Mesh by upgrading to HTTP 2 — Part 1

If you are using Istio and you are looking at easy ways to speed up the Service Mesh, this is the post for you.

Much has been written about how blazing fast HTTP 2 is verses HTTP 1.1 and how efficient it is. You can read about it here and here.

Most Service Mesh implementations I see still use the ancient HTTP 1.1 and do not upgrade its connections to HTTP/2.

However, a mesh is made up of many apps. Some of these apps could be using HTTP 1.1 or HTTP/2. In my experience, most apps(if not all) are still on HTTP 1.1. Upgrading all the microservices on the mesh could take some time and effort, even if the risks are neglibible.

If we upgrade all the mesh traffic to HTTP/2, we ought to get an immediate boost without taking any additional risk.

We can verify if our services are on HTTP1.1 or HTTP/2 by simply checking the istio-proxy logs.

To demonstrate this, I installed the Book Info Application which comes bundled with Istio.

Getting Started

Once you navigate to the Product page from your browser, open up the istio-proxy logs for both Product App and the downstream App — Details.

Product App (Before upgrade):

kubectl logs -f productpage-v1-64794f5db4-fqpgp -c istio-proxy
[2021-01-06T13:44:45.994Z] "GET /details/0 HTTP/1.1" 200 - "-" 0 178 2 1 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_6) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/13.0.5 Safari/605.1.15" "9a425a61-b88f-9b49-858d-252ed262e549" "details:9080" "10.1.1.31:9080" outbound|9080||details.default.svc.cluster.local 10.1.1.36:59852 10.99.56.191:9080 10.1.1.36:42486 - default
[2021-01-06T13:44:46.000Z] "GET /reviews/0 HTTP/1.1" 200 - "-" 0 379 15 15 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_6) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/13.0.5 Safari/605.1.15" "9a425a61-b88f-9b49-858d-252ed262e549" "reviews:9080" "10.1.1.34:9080" outbound|9080||reviews.default.svc.cluster.local 10.1.1.36:56232 10.101.5.33:9080 10.1.1.36:48422 - default
[2021-01-06T13:44:45.990Z] "GET /productpage HTTP/1.1" 200 - "-" 0 5183 28 28 "192.168.65.3" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_6) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/13.0.5 Safari/605.1.15" "9a425a61-b88f-9b49-858d-252ed262e549" "localhost" "127.0.0.1:9080" inbound|9080|| 127.0.0.1:50076 10.1.1.36:9080 192.168.65.3:0 outbound_.9080_._.productpage.default.svc.cluster.local default

Notice the default TCP protocol for both inbound and outbound connections are on HTTP 1.1.

Let’s check the logs for a downstream app.

Details App (Before upgrade):

kubectl logs -f details-v1-5974b67c8-rcnr9 -c istio-proxy
[2021-01-06T13:44:46.171Z] "GET /details/0 HTTP/1.1" 200 - "-" 0 178 1 1 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_6) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/13.0.5 Safari/605.1.15" "c00e80f6-bef2-9843-b4b9-eb1112ae3887" "details:9080" "127.0.0.1:9080" inbound|9080|| 127.0.0.1:49528 10.1.1.31:9080 10.1.1.36:59788 outbound_.9080_._.details.default.svc.cluster.local default

All we have to do now is to let Istio know that we want to upgrade all HTTP 1.1. calls to HTTP 2.

Note: There is another important secret step which I will reveal later.

Implementation

There are 2 ways to go about this:

Option 1: Upgrade the Global Mesh configurations to upgrade all HTTP 1.1 connections to HTTP/2

Option 2: Upgrade the destination rules 1 service at a time

We will explore Option 1 only in this post.

Option 1

This is simple to achieve. We have to patch the “istio” configmap in “istio-system” namespace with

h2UpgradePolicy: UPGRADE

Describe the current istio configmap in istio-system namespace.

kubectl describe configmap istio -n istio-system

Note down all the values under data/mesh.

Create configmap-patch.yaml

Add the above noted values under data/mesh to the below yaml file + h2UpgradePolicy: UPGRADE.

data:
  mesh: |-
    h2UpgradePolicy: UPGRADE
    accessLogFile: /dev/stdout
    defaultConfig:
      discoveryAddress: istiod.istio-system.svc:15012
      proxyMetadata:
        DNS_AGENT: ""
      tracing:
        zipkin:
          address: zipkin.istio-system:9411
    enablePrometheusMerge: true
    rootNamespace: istio-system
    trustDomain: cluster.local

Run the kubectl patch command.

kubectl patch configmap istio -n istio-system --patch "$(cat /configmap-patch.yaml)"

Describe the istio configmap in istio-system namespace again and verify the older values exist, in addition to the newly added one (h2UpgradePolicy: UPGRADE).

kubectl describe configmap istio -n istio-system

Note: Our intention is to add h2UpgradePolicy: UPGRADE and avoid tinkering with existing configurations.

That’s it!

Hit refresh on the Product Page like before.

Product App (After upgrade):

kubectl logs -f productpage-v1-64794f5db4-fqpgp -c istio-proxy
[2021-01-06T13:46:07.929Z] "GET /details/0 HTTP/1.1" 200 - "-" 0 178 6 6 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_6) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/13.0.5 Safari/605.1.15" "e1a591e7-2624-9616-b7b6-45d40346bcb4" "details:9080" "10.1.1.31:9080" outbound|9080||details.default.svc.cluster.local 10.1.1.36:33304 10.99.56.191:9080 10.1.1.36:43680 - default
[2021-01-06T13:46:07.940Z] "GET /reviews/0 HTTP/1.1" 200 - "-" 0 375 26 26 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_6) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/13.0.5 Safari/605.1.15" "e1a591e7-2624-9616-b7b6-45d40346bcb4" "reviews:9080" "10.1.1.35:9080" outbound|9080||reviews.default.svc.cluster.local 10.1.1.36:46090 10.101.5.33:9080 10.1.1.36:49620 - default
[2021-01-06T13:46:07.925Z] "GET /productpage HTTP/2" 200 - "-" 0 5179 45 44 "192.168.65.3" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_6) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/13.0.5 Safari/605.1.15" "e1a591e7-2624-9616-b7b6-45d40346bcb4" "localhost" "127.0.0.1:9080" inbound|9080|| 127.0.0.1:51270 10.1.1.36:9080 192.168.65.3:0 outbound_.9080_._.productpage.default.svc.cluster.local default

Wait! Why are some of the outbound logs still saying the protocol is HTTP 1.1, while the inbound logs are on HTTP/2?

This happens because the HTTP client libraries used by Product App in the example uses the HTTP 1.1 protocol. We need to use use libraries which support HTTP/2 (See Important Note below).

Details App (After upgrade):

kubectl logs -f details-v1-5974b67c8-rcnr9 -c istio-proxy
[2021-01-06T13:46:07.933Z] "GET /details/0 HTTP/2" 200 - "-" 0 178 2 1 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_6) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/13.0.5 Safari/605.1.15" "e1a591e7-2624-9616-b7b6-45d40346bcb4" "details:9080" "127.0.0.1:9080" inbound|9080|| 127.0.0.1:51276 10.1.1.31:9080 10.1.1.36:33304 outbound_.9080_._.details.default.svc.cluster.local default

The request which originated in Product App was on HTTP 1.1(see Product App logs after upgrade); Istio now upgrades it to HTTP/2 and passes it along to Details App.

Important Note

Remember, we are instructing Istio to upgrade the connections from HTTP 1.1 to HTTP/2. It cannot magicaly make the HTTP/ReST client libraries in deployed applications use HTTP/2. We need to upgrade our apps to use HTTP/2.

For example, Spring Boot supports HTTP/2, depending on the web server used. To move completely to HTTP/2, we need to instruct Spring Boot to move to HTTP/2.

Spring Boot Reference Documentation

Here is a helpful post:

Using HTTP/2 in your Spring Boot application

Conclusion

Its great to have Istio upgrade our connetions to a more efficient protocol. But, the upgrade should be avoided by moving all the apps on the mesh to HTTP/2. Considering, HTTP/2 has been around for a long time this shouldn’t be too hard whatever be the technology involved.

Risks are minimal, while the Benefits are tremendous.