How to resolve issues with Multiple Glue RDS Connections: 1 Trick That Works

This article discusses connection time-out errors that happen due to multiple connections in a Job. (Note: Each Connection is tested Successfully)

I came across this situation, where I required more than one connection in a job. The Task is simple, I need to extract the data from multiple RDS Instances, transform the data and load the enriched data into S3.

An AWS Glue Connection is a Data Catalog object that stores connection information for a particular data store. Connections store login credentials, URI strings, virtual private cloud (VPC) information, and more.

With respect to the above plan, I created two connections with connection type JDBC and tested the same (Each connection was successful). Then created a new job with the script and added those two connection to the job. When I ran the job, I was able to pull the data from first DB. It failed with below error, when it tried to pull the data from another db.

py4j.protocol.Py4JJavaError: An error occurred while calling o96.load.
: com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link failure

The last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server.
 at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
 at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
 at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
 at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
 at com.mysql.jdbc.Util.handleNewInstance(Util.java:403)
 at com.mysql.jdbc.SQLError.createCommunicationsException(SQLError.java:990)
 at com.mysql.jdbc.MysqlIO.<init>(MysqlIO.java:335)

On Debugging more, I came to know that AWS Glue supports one connection per job or development endpoint. If you specify more than one connection in a job, AWS Glue uses the first connection only.

In my case, Both RDS Instance is in the same VPC, with different security groups attached to each RDS Instance.

Follow the steps given below, to get the issue resolved.

1. Get the first connection name attached to the Glue Job.
2. Update the connection (From the step 1), with additional Security Group which belongs to another RDS Instance.

Thank you for your support and don’t forget to 👏👏

Read all about our Amazon web service practice