Register external services containing paths in endpoints into Consul Service Mesh

Problem

When you deploy an application into a Consul Service Mesh it will deploy together with sidecar envoy proxy that handles your network traffic. This application can only access services inside mesh. One way to access any service outside the mesh is to register it using a Terminating Gateway.

It some cases the external application you want to register into Service Mesh can be hosted behind another proxy under specific path i.e your-proxy.domain.com/path/to/service. It may seem to be pretty easy at first glance but putting the whole address into the Consul Service address field will not work and cause a No health upstream error message.

Sometimes there is a general misunderstanding on how envoy proxy servers route traffic. If you are attempting to route the incoming path / to a backed service path of /some/path, this is not how it works. Envoy is a L3/L4 network proxy so it is not able to interpret paths and your incoming endpoint will be passed along to the backend service without any paths.

Prerequisites

In order to test this solution you need to have:

• Consul Service Mesh
• Nomad Cluster
• Network connectivity to external application
• Terminating Gateway deployed in Service Mesh

Solution

One solution is to use the Consul KV store and the Nomad job templating mechanism.

We will create a new environment variable for mesh service to use it as an endpoint to an external service.

There are multiple ways of interacting with Consul API. In this article we will use Terraform.

Registering service

It is vital to create a Consul KV store entry along with a Consul Service as we will use it in next steps.

resource “consul_service” “this” {
  name = var.external_service.name
  node = "<consul_node_name>"
  port = var.external_service.port
  
  check {
    check_id        = var.external_service.name
    name            = “${var.external_service.name} health check”
    status          = “passing”
    http            = “https://${var.external_service.addresses}${var.external_service.path}${var.external_service.health_check_path}"
    tls_skip_verify = false
    interval        = “30s”
    timeout         = “3s”
  }
}

resource "consul_key" "this" {
  datacenter = "<your-consul-data-center>"
  
  key {
    path  = "external-services/${var.external_service.name}/path_to_service
    value = var.external_service.path
  }
}

Consuming endpoints in the mesh

Once we have our service registered and key in the Consul KV store created we can make use of it in our Nomad job.

To make a connection possible we need to first register the external service as an upstream in our mesh. We can pass a list of upstreams to our Nomad job definition.

variable "upstreams" {
  type    = list(string)
  default = ["external-service-a", "external-service-b"]
}

resource "nomad_job" "this" {
  ...
  hcl2 {
    ...
    upstreams = var.upstreams
    ...
  }
}

In our job.nomad.hcl we will use the Nomad job templating mechanism and Nomad built-in environment variables to create a new external service endpoint.

variable "upstreams" {
  type = list(string)
}

locals {
  upstream_starting_port = 12345
  upstreams              = join(" ", [for upstream in upstreams : "\"${upstream}\""])
}

job "name" {
  ...
  group "name" {
    service "name" {
      dynamic "upstreams" {
        for_each = [for index, upstream in var.upstreams : {
          destination_name = upstream
          local_bind_port  = (local.upstream_starting_port + index)
        }]
  
        content {
          destination_name = upstreams.value.destination_name
          local_bind_port  = upstreams.value.local_bind_port
        }
      }
    }
    ...
    task "name" {
      ...
      template {
        data = <<EOF
{{- range $u := sprig_list ${local.upstreams} }}
EXTERNAL_APPLICATION_URL_{{ $u | replaceAll "-" "_" }}=http://{{ $u | sprig_cat "NOMAD_UPSTREAM_ADDR_" | sprig_nospace | env }}{{ keyOrDefault (sprig_cat "external-services/" $u "/path_to_service" | sprig_nospace) "" }}
{{- end -}}
        EOF
        destination = "local/env_vars"
        env         = true
      }
    }
  }
}   

We had to create an additional local.upstreams variable to fit the format of the list that is required by Go templating.

Next we iterated over our list of upstreams and we templated new environment variables that use Nomad Runtime environment variable NOMAD_UPSTREAM_ADDR_<service_name>.

We also used a several Consul Templating functions to manipulate data such as:

• range
• env
• replaceAll
• sprig_list
• sprig_cat
• sprig_nospace
• keyOrDefault

Have you come across this before? What solution have you used? If not, I hope it was helpful for you.