Using ElastiCache with AWS RDS

Amazon Relational Database Service, or RDS, is a managed database service that helps you quickly create relational databases and automates backups and failovers so you don't have to wake up in the middle of the night to bring a database back up. ElasticCache is an in-memory caching that provides microsecond read and write latency. ElastiCache is compatible with Redis and Memcached. We are going to be using Redis with ElasticCache.

Let's get started. If you don't already have a VPC, use the SimpleVPC Terraform code by going to GitHub and cloning the Simple-VPC repository. Update the variables.tf and change the ENV on line 32. The terraform code will create a VPC with six Subnets, three public and three private. It will create an IGW with three NATs.

git clone https://github.com/mahmoodr786/simple-vpc.git
cd simple-vpc
terraform init
terraform apply

Now that we have our VPC, let's create our RDS, but before we do that, we need to create a Subnet Group. Go to the RDS console and click on subnet groups, then click on create DB subnet group.

Select your VPC and select your availability zones.

Select your private subnets and click Create. Note you should always keep databases in private subnets and not publicly accessible to protect data.

Now, let's create our Aurora (MySQL-compatible) RDS DB.

We will use production as the template and put a DB identifier. Pick a strong password. You can also use Auto-generate a password or AWS Secrets Manager to generate a strong password.

I will go with Aurora Standard, and a db.r5.large should be more than enough. We don't need a Multi-AZ replica, as we are using the DB for testing. Still, in production, where you have more critical data, it is highly advised to do Multi-AZ deployment for availability.

Keep the default settings for Compute resource and network type. Select your VPC and your subnet group.

As you can see below, AWS attaches the default security group when you create services. You should not use the default security group because you don't want to allow access to services by accident. Keep everything else as it is.

For our testing, we will not need or use the monitoring and database authentication using IAM or Kerberos. We don't need to turn on read replica write forwarding as we only have one DB.

Expand the Additional configuration and type in an initial database name. If you are doing this for real production data, it is recommended that you set your backup to at least seven days and ensure you always enable encryption using KMS.

We don't need to backtrack or Log exports for our testing, but you should enable them for the production workload.

Finally, we get an estimate of how much this DB will cost us, and you can click on Create Database.

ElastiCache

While our database is being created, we can create our ElastiCache cluster.

As you can see below, RDS already tells us that we can see 55% in cost, and our workload will be 80x faster. Select Redis for the cluster; we are not using the cluster mode for exercise. Name your cache, and you can use three replicas for high availability. A cache.t3.medium should be enough for our testing, but you should plan how much data you will keep in memory and create a cluster for that size.

We also need to create a subnet group for ElastiCache and select our private subnets.

Ensure encryption at rest and transit is enabled, review everything, and then click Create ElastiCache Cluster.

Testing

We will use some basic Python code to query 210k rows of data and dump them into JSON, and we will use the Linux time command to see how long it takes. We will repeat the same test by using the ElastiCache as the front and see how much faster it gets. Obviously, this is not true benchmarking, but it should be enough to give us an idea of the speed.

The code to create our random table:

CREATE TABLE `random` (
  `id` int(11) unsigned NOT NULL AUTO_INCREMENT,
  `name` varchar(255) DEFAULT NULL,
  `email` varchar(255) DEFAULT NULL,
  `address` varchar(255) DEFAULT NULL,
  `phone` varchar(255) DEFAULT NULL,
  `job_title` varchar(255) DEFAULT NULL,
  `date_of_birth` varchar(255) DEFAULT NULL,
  `info` text DEFAULT NULL,
  PRIMARY KEY (`id`)
) ENGINE=InnoDB;

We will use Faker to insert some fake data on that table. Install Faker, redis, and MySQL connector.

pip install redis mysql-connector-python faker

Run our faker code:

from faker import Faker
import mysql.connector

fake = Faker()

# Connect to your MySQL database
connection = mysql.connector.connect(
    host="data1.cluster-cifbe2qttmpd.us-east-1.rds.amazonaws.com",
    user="admin",
    password="***********",
    database="data1"
)
cursor = connection.cursor()

# Insert random data into your MySQL table
for _ in range(210000):
    name = fake.name()
    email = fake.email()
    address = fake.address()
    job_title = fake.job()
    date_of_birth = fake.date_of_birth(minimum_age=18, maximum_age=65).strftime('%Y-%m-%d')
    info = fake.text()

    cursor.execute("INSERT INTO random (id, name, email, address, job_title, date_of_birth, info) "
                   "VALUES (null, %s, %s, %s, %s, %s, %s)",
                   (name, email, address, job_title, date_of_birth, info))

# Commit the changes and close the connection
connection.commit()
cursor.close()
connection.close()

Now, dump data from MySQL to a file using the database. This script will also cache the data on the Redis, so when we rerun it, it should be faster. Run the script below

import redis
import mysql.connector
import ssl
import json

# ElastiCache (Redis) Connection
redis_host = 'master.data1-cache1.3zri6h.use1.cache.amazonaws.com'
redis_port = 6582

ssl_context = ssl.create_default_context()
ssl_context.check_hostname = False
ssl_context.verify_mode = ssl.CERT_NONE

# Connect to ElastiCache without SSL verification
redis_client = redis.StrictRedis(
    host=redis_host,
    port=redis_port,
    decode_responses=True,
    ssl=ssl_context
)

mysql_conn = mysql.connector.connect(
    host="data1.cluster-cifbe2qttmpd.us-east-1.rds.amazonaws.com",
    user="admin",
    password="**************",
    database="data1"
)

mysql_cursor = mysql_conn.cursor()

key = "AllMySQLData"
data = redis_client.get(key)
if not data:
    mysql_cursor.execute("SELECT * FROM random")
    data = mysql_cursor.fetchall()
    data = json.dumps(data)
    redis_client.set(key, data)
print(data)

mysql_cursor.close()
mysql_conn.close()
redis_client.connection_pool.disconnect()

We are getting about 1.5 seconds going through MySQL.

Now, through ElastiCache, we are getting about 1.1 seconds, so a 0.4-second improvement. That is about a 27% improvement in speed.

That is it for this blog. Delete all the stuff you have created manually and call a terraform destroy.