Register external services containing paths in endpoints into Consul Service Mesh
Problem
When you deploy an application into a Consul Service Mesh it will deploy together with sidecar envoy proxy that handles your network traffic. This application can only access services inside mesh. One way to access any service outside the mesh is to register it using a Terminating Gateway.
It some cases the external application you want to register into Service Mesh can be hosted behind another proxy under specific path i.e your-proxy.domain.com/path/to/service
. It may seem to be pretty easy at first glance but putting the whole address into the Consul Service address field will not work and cause a No health upstream
error message.
Sometimes there is a general misunderstanding on how envoy proxy servers route traffic. If you are attempting to route the incoming path /
to a backed service path of /some/path
, this is not how it works. Envoy is a L3/L4 network proxy so it is not able to interpret paths and your incoming endpoint will be passed along to the backend service without any paths.
Prerequisites
In order to test this solution you need to have:
- Consul Service Mesh
- Nomad Cluster
- Network connectivity to external application
- Terminating Gateway deployed in Service Mesh
Solution
One solution is to use the Consul KV store and the Nomad job templating mechanism.
We will create a new environment variable for mesh service to use it as an endpoint to an external service.
There are multiple ways of interacting with Consul API. In this article we will use Terraform.
Registering service
It is vital to create a Consul KV store entry along with a Consul Service as we will use it in next steps.
resource “consul_service” “this” {
name = var.external_service.name
node = "<consul_node_name>"
port = var.external_service.port
check {
check_id = var.external_service.name
name = “${var.external_service.name} health check”
status = “passing”
http = “https://${var.external_service.addresses}${var.external_service.path}${var.external_service.health_check_path}"
tls_skip_verify = false
interval = “30s”
timeout = “3s”
}
}
resource "consul_key" "this" {
datacenter = "<your-consul-data-center>"
key {
path = "external-services/${var.external_service.name}/path_to_service
value = var.external_service.path
}
}
Consuming endpoints in the mesh
Once we have our service registered and key in the Consul KV store created we can make use of it in our Nomad job.
To make a connection possible we need to first register the external service as an upstream in our mesh. We can pass a list of upstreams to our Nomad job definition.
variable "upstreams" {
type = list(string)
default = ["external-service-a", "external-service-b"]
}
resource "nomad_job" "this" {
...
hcl2 {
...
upstreams = var.upstreams
...
}
}
In our job.nomad.hcl
we will use the Nomad job templating mechanism and Nomad built-in environment variables to create a new external service endpoint.
variable "upstreams" {
type = list(string)
}
locals {
upstream_starting_port = 12345
upstreams = join(" ", [for upstream in upstreams : "\"${upstream}\""])
}
job "name" {
...
group "name" {
service "name" {
dynamic "upstreams" {
for_each = [for index, upstream in var.upstreams : {
destination_name = upstream
local_bind_port = (local.upstream_starting_port + index)
}]
content {
destination_name = upstreams.value.destination_name
local_bind_port = upstreams.value.local_bind_port
}
}
}
...
task "name" {
...
template {
data = <<EOF
{{- range $u := sprig_list ${local.upstreams} }}
EXTERNAL_APPLICATION_URL_{{ $u | replaceAll "-" "_" }}=http://{{ $u | sprig_cat "NOMAD_UPSTREAM_ADDR_" | sprig_nospace | env }}{{ keyOrDefault (sprig_cat "external-services/" $u "/path_to_service" | sprig_nospace) "" }}
{{- end -}}
EOF
destination = "local/env_vars"
env = true
}
}
}
}
We had to create an additional local.upstreams
variable to fit the format of the list that is required by Go templating.
Next we iterated over our list of upstreams and we templated new environment variables that use Nomad Runtime environment variable NOMAD_UPSTREAM_ADDR_<service_name>.
We also used a several Consul Templating functions to manipulate data such as:
- range
- env
- replaceAll
- sprig_list
- sprig_cat
- sprig_nospace
- keyOrDefault
Have you come across this before? What solution have you used? If not, I hope it was helpful for you.