Open in app

Sign in

Write

Sign in

Multi-Region Applications With Kong Mesh and YugabyteDB

Denis Magda
11 min readSep 18, 2023

--

Multi-region applications are becoming more widespread these days. Some create multi-region apps to tolerate region-level outages, some to comply with data regulatory requirements by pinning user data to specific cloud regions, and some to achieve low latency for all users regardless of their location.

This guide shows how to build multi-region apps with Kong Mesh and YugabyteDB, focusing on low latency. We’ll create a service for a pizza company that bakes and delivers pizzas to customers living in three cities across three continents: New York in North America, Berlin in Europe, and Sydney in Australia. The final app will handle read and write requests with low latency across all locations.

Reference Architecture

The pizza app comprises two microservices: Kitchen and Tracker. The Kitchen microservice is used to place a new pizza order in the queue, while the Tracker microservice is utilized to check the status of a pizza order.

If we aim for both microservices to handle user requests with low latency across New York, Berlin, and Sydney, then, at a minimum, a microservice instance has to be deployed near those cities. The full reference architecture can be outlined as follows:

First, the instances of the Kitchen and Tracker microservices are deployed across three cloud regions: US East4 in N. Virginia, Europe West3 in Germany, and Australia Southeast1 near Sydney.

Next, Kong Mesh is used as a service mesh to simplify the deployment and management of our microservices in a multi-region setting. Each cloud region will have the following Kong components:

  • KUMA-CP — a region-level Kuma Control Plane (CP) responsible for the configuration and management of the region-level data planes.
  • KUMA-DP — a region-level Data Plane (DP) that consists of the proxies running alongside our microservice. All mesh traffic between the services flows through these proxies.
  • Mesh Gateway — a built-in Kong Mesh gateway with a dedicated DP that handles external traffic coming from web frontends, mobile apps, and other endpoints.

Besides these components, a multi-region Kong Mesh deployment requires a Global Control Plane (CP) to manage the policies of the entire mesh. This instance is deployed in the US East4 region.

Lastly, YugabyteDB is used as a distributed Postgres-compatible database that can automatically pin pizza orders and other user data to specific geographic locations. The database nodes are provisioned in the same cloud regions where we have Kong Mesh components and microservice instances running.

Step 1: Deploy a Geo-Partitioned Database Cluster

Now, let’s bring this reference architecture to life, starting with the database.

YugabyteDB supports a geo-partitioned deployment mode that allows placing user data in required locations. This is precisely what we need to achieve low latency requests for all the customers of our pizzeria operating in New York, Berlin, and Sydney.

The fastest way to bootstrap a geo-partitioned cluster is by using YugabyteDB Managed:

Simply head to YugabyteDB Managed and initiate a dedicated partitioned cluster with nodes in N. Virginia (us-east4), Frankfurt (europe-west3), and Sydney (australia-southeast1).

Once the database is set up, execute the following script to create database objects used by the Kitchen and Tracker microservices. Among those database objects, the most noteworthy ones related to how the database utilizes the partitioning feature of Postgres to pin pizza orders to specific locations are:

CREATE TABLE pizza_order(
    id int,
    status order_status NOT NULL,
    location store_location NOT NULL,
    order_time timestamp NOT NULL DEFAULT now()
)
PARTITION BY LIST (location);

CREATE TABLE pizza_order_usa PARTITION OF pizza_order(id, status, location, order_time, PRIMARY KEY (id, location))
FOR VALUES IN ('NewYork') TABLESPACE usa_ts;

CREATE TABLE pizza_order_europe PARTITION OF pizza_order(id, status, location, order_time, PRIMARY KEY (id, location))
FOR VALUES IN ('Berlin') TABLESPACE europe_ts;

CREATE TABLE pizza_order_australia PARTITION OF pizza_order(id, status, location, order_time, PRIMARY KEY (id, location))
FOR VALUES IN ('Sydney') TABLESPACE australia_ts;
  • The pizza_order table is partitioned by the location column.
  • Depending on the value of the location, the record will be written to or queried from a specific tablespace that resides in one of the customer geographies.

Step 2: Start VMs in the Cloud

Kong Mesh can be deployed on Kubernetes, VMs, or bare metal. We’ll use Kong’s Universal deployment mode to get the multi-region pizza service running on VMs.

The subsequent steps are:

  • Start VMs in the cloud regions: us-east4, europe-west3, and australia-southeast1, close to the pizza customers. I’m using Google Cloud Platform, but you’re free to use any public cloud of your choice.
  • Configure the firewall to allow HTTP requests on ports: 8080, 5681, 5685, and 6681. Port 8080 will be accessed by customer-facing apps, while the other ports are specific to Kong Mesh.

For each VM:

  • Install JDK 17 or higher.
  • Install Kong Mesh.
curl -L https://docs.konghq.com/mesh/installer.sh | VERSION=2.4.0 sh -

cd kong-mesh-2.4.0/bin
PATH=$(pwd):$PATH
  • Clone the pizza app project:
git clone https://github.com/YugabyteDB-Samples/pizza-store-kong-mesh.git
  • Go to the project root directory and create a folder for the logs:
mkdir logs

Step 3: Deploy Global Control Plane

The Global Control Plane (CP) is utilized to push down mesh-related settings and policies to all the regional CPs.

  • SSH into the VM in us-east4 that will host the instance of the Global CP.
  • Specify a path to the Kong Mesh license file (required if you have more than 6 data planes, as is the case with the pizza service).
export KMESH_LICENSE_PATH={path_to_the_license_file}
  • Use kuma-cp to initiate the Global CP instance:
cd pizza-store-kong-mesh

KUMA_MODE=global \
KUMA_DIAGNOSTICS_SERVER_PORT=6680 \
KUMA_API_SERVER_HTTP_PORT=6681 \
KUMA_API_SERVER_HTTPS_PORT=6682 \
KUMA_INTER_CP_SERVER_PORT=6683 \
kuma-cp run > logs/global-cp.log 2>&1 &
  • Open http://{VM_IP_ADDRESS}:6681/gui to verify that the plane is active and operational.

We will use the same VM in us-east4 for the regional CP. Therefore, register the Global CP context on this VM separately and set up the initial global policies for all locations:

  • Register the Global CP:
kumactl config control-planes add --address http://localhost:6681 --name global-cp
  • Define policies and routes for the mesh gateways:
cd pizza-store-kong-mesh

kumactl apply -f standalone/mesh-gateway-config.yaml
kumactl apply -f standalone/mesh-gateway-route-config.yaml
  • Switch kumactl back to the local/regional context:
kumactl config control-planes switch --name local

Step 4: Create Regional Control Planes

Kong regional Control Planes (CPs) are tasked with configuring regional data plane proxies, ingress, and other region-specific settings.

To deploy an instance of the regional CP, execute the following command on every VM:

cd pizza-store-kong-mesh

KUMA_MODE=zone \
KUMA_MULTIZONE_ZONE_NAME={ZONE_NAME} \
KUMA_MULTIZONE_ZONE_KDS_TLS_SKIP_VERIFY=true \
KUMA_MULTIZONE_ZONE_GLOBAL_ADDRESS=grpcs://{GLOBAL_CP_IP_ADDRESS}:5685 \
kuma-cp run > logs/zone-cp.log 2>&1 &
  • {ZONE_NAME} - represents a unique name for the zone/region. Depending on the actual VM region, use us-east4, europe-west3, or australia-southeast1.
  • {GLOBAL_CP_IP_ADDRESS} - refers to the public IP address of the VM where the Global CP operates (for this pizza service, it's the us-east4 VM).

After deploying all the CPs, navigate to the Global CP UI to ensure they’ve successfully registered with the mesh: http://{GLOBAL_CP_IP_ADDRESS}:6681/gui/zones/zone-cps

Step 5: Deploy Services and Data Planes

It’s now time to deploy the microservice instances along with their data planes (DPs) in all the selected cloud locations.

Follow the steps below on each VM:

  1. Set the database connectivity settings using the environment variables:
export DB_URL="{db_url}"
export DB_USER={db_user}
export DB_PASSWORD={db_password}

The DB_URL should follow this format: jdbc:postgresql://{REGIONAL_HOSTNAME}:5433/yugabyte. You can locate the {REGIONAL_HOSTNAME} in the Connection Parameters section of the YugabyteDB Managed UI.

Ensure that each VM connects to a database endpoint located in the same region. For example, a VM in europe-west3 should connect to the database endpoint in europe-west3.

2. Deploy the Kitchen and Tracker instances in every region.

cd pizza-store-kong-mesh/kitchen

mvn spring-boot:run > ../logs/kitchen-service.log 2>&1 &

cd pizza-store-kong-mesh/tracker

mvn spring-boot:run > ../logs/tracker-service.log 2>&1 &

3. Deploy DPs for each microservice.

cd pizza-store-kong-mesh

kumactl generate dataplane-token \
    --tag kuma.io/service=kitchen-service \
    --valid-for=720h > /tmp/kuma-token-kitchen-service

kuma-dp run \
    --cp-address=https://localhost:5678 \
    --dataplane-file=standalone/kitchen-dp-config.yaml \
    --dataplane-token-file=/tmp/kuma-token-kitchen-service \
    > logs/kitchen-dp.log 2>&1 &

kumactl generate dataplane-token \
    --tag kuma.io/service=tracker-service \
    --valid-for=720h > /tmp/kuma-token-tracker-service

kuma-dp run \
    --cp-address=https://localhost:5678 \
    --dataplane-file=standalone/tracker-dp-config.yaml \
    --dataplane-token-file=/tmp/kuma-token-tracker-service \
    > logs/tracker-dp.log 2>&1 &

After deploying all the services and DPs, navigate to the Global CP UI to verify their registration with the mesh: http://{GLOBAL_CP_IP_ADDRESS}:6681/gui/mesh/default/data-planes

Step 6: Set Up Gateways

The gateway is the remaining component that needs deployment in each cloud region. While DPs handle inter-service communication within the mesh, gateways manage the external traffic entering the mesh. All user requests from the web, mobile, or other endpoints will reach the services through these gateways.

Deploy a gateway instance on each VM:

cd pizza-store-kong-mesh

kumactl generate dataplane-token \
    --tag kuma.io/service=mesh-gateway \
    --valid-for=720h > /tmp/kuma-token-mesh-gateway

kuma-dp run \
    --cp-address=https://localhost:5678/ \
    --dns-enabled=false \
    --dataplane-token-file=/tmp/kuma-token-mesh-gateway \
    --dataplane-file=standalone/mesh-gateway-dp-config.yaml \
    > logs/gateway.log 2>&1 &

We configured the gateway routes earlier via the Global CP instance in step 3, which are as follows:

type: MeshGatewayRoute
mesh: default
name: mesh-gateway-default-route
selectors:
  - match:
      kuma.io/service: mesh-gateway
conf:
  http:
    rules:
      - matches:
          - path:
              match: PREFIX
              value: /kitchen
        backends:
          - destination:
              kuma.io/service: kitchen-service
      - matches:
          - path:
              match: PREFIX
              value: /tracker
        backends:
          - destination:
              kuma.io/service: tracker-service

This configuration instructs the gateway instances to forward user requests to the local service instances based on the path value. For example, if the path starts with /kitchen, the request will be directed to the Kitchen service.

Finally, visit the Global CP UI to confirm all gateways have registered with the mesh: http://{GLOBAL_CP_IP_ADDRESS}:6681/gui/mesh/default/gateways

Confirming Low Latency Across All Regions

Having implemented this multi-region architecture and started serving pizzas to customers in New York, Berlin, and Sydney, we must ask: How will latency differ among these locations? Will the service perform faster in New York compared to Sydney?

Let’s find out.

For instance, consider customers in New York using their phones to order pizzas. Their requests will be directed to the gateway instance in us-east4 and be processed by microservice instances in the same region.

To simulate this scenario, SSH into the VM in us-east4 and execute the following command several times, incrementing the id with each run:

time http POST :8080/kitchen/order location==NewYork id==3

Upon invoking the Kitchen service, the following response is expected. The db latency attribute indicates the actual latency between the backend and the database, which is typically the most time-consuming segment of the request journey:

{
    "db latency": "0.006s",
    "result: ": {
        "id": 3,
        "location": "NewYork",
        "orderTime": "2023-09-18T20:08:27.967+00:00",
        "status": "Ordered"
    }
}


real 0m0.311s
user 0m0.270s
sys 0m0.024s

To check the order’s status, send the subsequent request to the gateway in us-east4, and it will be processed by the local Tracker instance:

time http GET :8080/tracker/order location==NewYork id==3

...
{
    "db latency": "0.004s",
    "result: ": {
        "id": 3,
        "location": "NewYork",
        "orderTime": "2023-09-18T20:08:27.968+00:00",
        "status": "Ordered"
    }
}


real 0m0.295s
user 0m0.260s
sys 0m0.024s

Overall, let’s record these latency figures:

New York:

  • New Order (writes): 311 ms (total latency), 6 ms (db latency)
  • Order Status (reads): 295 ms (total latency), 4 ms (db latency)

It’s important to note that the total latency will vary based on your hardware and can change greatly. In this instance, I implemented the solution on low-end VMs equipped with 2 vCPUs.

Subsequently, perform the same tests for customers in Berlin and Sydney:

  • Create several pizza orders and check their statuses from the europe-west3 VM.
time http POST :8080/kitchen/order location==Berlin id==12

...
{
    "db latency": "0.009s",
    "result: ": {
        "id": 12,
        "location": "Berlin",
        "orderTime": "2023-09-18T20:32:53.219+00:00",
        "status": "Ordered"
    }
}


real 0m0.375s
user 0m0.279s
sys 0m0.076s

time http GET :8080/tracker/order location==Berlin id==12

...
{
    "db latency": "0.003s",
    "result: ": {
        "id": 12,
        "location": "Berlin",
        "orderTime": "2023-09-18T20:32:53.221+00:00",
        "status": "Ordered"
    }
}


real 0m0.381s
user 0m0.308s
sys 0m0.032s
  • Repeat the same steps from the australia-southeast1 VM.
time http POST :8080/kitchen/order location==Sydney id==22

...
{
    "db latency": "0.008s",
    "result: ": {
        "id": 22,
        "location": "Sydney",
        "orderTime": "2023-09-18T20:36:13.681+00:00",
        "status": "Ordered"
    }
}


real 0m0.305s
user 0m0.261s
sys 0m0.028s

time http GET :8080/tracker/order location==Sydney id==22

...
{
    "db latency": "0.004s",
    "result: ": {
        "id": 22,
        "location": "Sydney",
        "orderTime": "2023-09-18T20:36:13.682+00:00",
        "status": "Ordered"
    }
}


real 0m0.305s
user 0m0.273s
sys 0m0.021s

Let’s compile all the latency figures to confirm that they remain consistent and low across all customer locations:

New York:

  • New Order (writes): 311 ms (total latency), 6 ms (db latency)
  • Order Status (reads): 295 ms (total latency), 4 ms (db latency)

Berlin:

  • New Order (writes): 375 ms (total latency), 9 ms (db latency)
  • Order Status (reads): 381 ms (total latency), 3 ms (db latency)

Sydney:

  • New Order (writes): 305 ms (total latency), 8 ms (db latency)
  • Order Status (reads): 305 ms (total latency), 4 ms (db latency)

Job done! We’ve successfully built a multi-region application for a pizza company that processes user requests with minimal latency, irrespective of user location.

Bonus: Cross-Region Requests Out-of-the-Box

A noteworthy advantage of this solution is its ability to handle cross-region requests seamlessly.

Consider a scenario where a customer from Berlin is on a business trip in Sydney. His family back in Berlin wants him to order a pizza for them. Here’s what happens when he places the order:

  • The customer opens the pizza app, and the app routes the order to the gateway instance in the australia-southeast1 region due to the customer’s current location.
  • This gateway then directs the request to the Kitchen service within the same region.
  • The service inserts the order in the database, sending a request to a database node in Australia.
  • Recognizing that the location is set to Berlin, the database node from Australia sends the order to database nodes located in Europe.

While the latency will increase due to the intercontinental request journey from Australia to Europe, this entire process is transparent, requiring no modifications on either the Kong Mesh or application side.

#executing on the VM in Sydney

time http POST :8080/kitchen/order location==Berlin id==34

...
{
    "db latency": "0.991s",
    "result: ": {
        "id": 34,
        "location": "Berlin",
        "orderTime": "2023-09-18T20:55:05.956+00:00",
        "status": "Ordered"
    }
}


real 0m1.295s
user 0m0.272s
sys 0m0.025s

Conclusion

Multi-region apps let you benefit from the global cloud infrastructure by building solutions that perform with low latency across regions, withstand various cloud outages, and comply with data regulatory requirements by pinning user data to specific data centers.

Want to learn more? See how to use Global Cloud Load Balancers to automatically route user requests to the nearest application and gateway instances.

Cloud
API
Database
Yugabytedb
Kong

--

--

Denis Magda

Written by Denis Magda

179 Followers

Distributed applications, systems and databases

Help

Status

About

Careers

Press

Blog

Privacy

Terms

Text to speech

Teams