Kubernetes Pod network connectivity issue with Istio enabled
A K8s Pod can return an Error when trying to reach the network before the Istio Sidecar is Running.
Many applications execute commands or checks during startup, which require network connectivity. This can cause application containers to hang or restart if the istio-proxy
sidecar container is not ready.
This article demonstrates how to use Istio’s annotation holdApplicationUntilProxyStarts
to avoid network connectivity issues.
Test Environment
The test environment had the following tech stack:
- macOS Monterey version 12.3.1
- Apple M1 Pro
- minikube version v1.26.0
- K8s version v1.22.7
- Istio version 1.13.3
Check the official documentation on how to install Istio
Before you begin
1. Start minikube
with the specific K8s version:
minikube start — kubernetes-version=v1.22.7
2. Deploy the httpbin
sample application:
- Make sure your current directory is the
istio
directory. - Start the
httpbin
sample.
kubectl apply -f samples/httpbin/httpbin.yaml
3. Review the httpbin
deployment:
$ kubectl get services httpbin
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
httpbin ClusterIP 10.105.138.79 <none> 8000/TCP 32d$ kubectl get pods httpbin-74fb669cc6-4qs24
NAME READY STATUS RESTARTS AGE
httpbin-74fb669cc6-4qs24 2/2 Running 0 32d
Scenario 1 — Pod with Error
Now that Istio and httpbin
are deployed and working we can deploy our Pod pod-test-connectivity
. This is the Pod yaml
file:
apiVersion: v1
kind: Pod
metadata:
name: "pod-test-connectivity"
spec:
containers:
- name: wget
image: busybox
command: ['wget']
args: [ '-qO-', 'httpbin:8000']
restartPolicy: Never
This Pod will run a wget
command to GET
the httpbin
application.
Create the Pod:
$ kubectl apply -f pod-connectivity.yaml
pod/pod-test-connectivity created
Check Pod status:
$ kubectl get pods pod-test-connectivity
NAME READY STATUS RESTARTS AGE
pod-test-connectivity 1/2 Error 0 25s
The Pod is in Error
status. Looking at the logs we can see:
$ kubectl logs pod-test-connectivity
wget: can't connect to remote host (10.105.138.79): Connection refused
Delete the Pod:
$ kubectl delete -f pod-connectivity.yaml
pod/pod-test-connectivity deleted
Scenario 2 — Pod Running Successfully
This time, we will add the Istio annotation to the Pod manifest file:
apiVersion: v1
kind: Pod
metadata:
name: "pod-test-connectivity"
annotations:
proxy.istio.io/config: '{ "holdApplicationUntilProxyStarts": true }'
spec:
containers:
- name: curl
image: curlimages/curl
command: ["/bin/sh"]
args:
- "-c"
- |
curl httpbin:8000
curl -s -X POST 127.0.0.1:15000/quitquitquit #to exit the istio-proxy sidecar
restartPolicy: Never
Create the Pod:
$ kubectl apply -f pod-connectivity.yaml
pod/pod-test-connectivity created
Check Pod status:
$ kubectl get pods pod-test-connectivity
NAME READY STATUS RESTARTS AGE
pod-test-connectivity 0/2 Completed 0 7s
Now our Pod completed the curl
without any problems. Check the logs to confirm:
$ kubectl logs pod-test-connectivity...
<div class="wrapper">
<section class="block col-12 block-desktop col-12-desktop">
<div><h2>Other Utilities</h2><ul>
<li>
<a href="/forms/post">HTML form</a> that posts to /post /forms/post</li>
</ul><br />
<br />
</div>
</section>
</div>
</div>
</body></html>OK
Using the annotation holdApplicationUntilProxyStarts
prevents the Pod to execute any commands before the istio-proxy
is Running
. Therefore, when the Pod executes the curl
it already has network connectivity.