Problems when trying to connect AWS Services with Istio

Have your services inside Istio service mesh are connect to AWS RDS with correct username and password but answer you back with incorrect username/password response? If your services need to connect multiple RDS with different username and password, here maybe the reason: ServiceEntry, your connections are different than you expected.

Few days ago, I am trying to deploy our services into service mesh (Istio) and faced some problems(maybe not) when tried to connect AWS RDS. Get a response like:

mysqli_real_connect(): (HY000/1045): Access denied for user 'user'@'your.ip.address' (using password: YES)

Looks like the service is connected, but failed in authentication part. Why?

I did a little experiment with phpmyadmin inside service mesh to figure it out. Let’s assume we have 2 database, d1 and d2. Each database have specific username/password:

• Connect to d1 with username1/password1: It’s working
• Connect to d2 with username2/password2: Not working
• Connect to d2 with username1/password1: IT’S WORKING!

Clearly, the service is connecting to the wrong database. But why?

As we know, if your traffic will go outside the mesh, you need to specify an ServiceEntry as following:

However, if services are connecting to multiple RDS or Redis, your ServiceEntry may look like:

Looks good right? But here is the problem, resolution: DNS

Why? What happened to resolution: DNS? The istio provides 3 methods for the resolution: NONE, STATIC and DNS. The major difference between those three is, when will you know the exactly address of your request/traffic. After that, I realized one thing, we don’t need to get ip address from our proxy server. Just leave your DNS to AWS. So, the ServiceEntry will look like:

Problem solved.