Kubernetes — Microservices Discovery

Photo by N. on Unsplash

In non-Kubernetes-based projects, we have a discovery server (like the Eureka service) to take care of service discovery. So I was curious to know which component we should use in Kubernetes. I did search on internet to check if any article covers discovery with upstream microservice calling downstream microservice and found none; hence thought to write this article.

The following are the prerequisites:

• Spring Boot version: 2.7.3
• Minikube version: v1.25.1
• Kubectl: v1.22.5

In this article, I will explain how microservices discover each other in the Kubernetes (K8s) environment. I will demo it using SpringBoot-based microservices (Upstream calls downstream using a Feign Client).

How does Kubernetes supports discovery?

Discovery in Kubernetes is supported in two ways.

1. Environment variables

Kubernetes adds {SVCNAME}_SERVICE_HOST and {SVCNAME}_SERVICE_PORT environment variables for each pod created in a cluster. The following is an example.

PAYMENT_SERVICE_SERVICE_HOST=10.108.155.131
PAYMENT_SERVICE_SERVICE_PORT=8080
TICKET_SERVICE_SERVICE_PORT=8080
TICKET_SERVICE_SERVICE_HOST=10.108.155.131

In Feign Client, use these variables to make service URLs, as shown below.

url = ${PAYMENT_SERVICE_SERVICE_HOST}:${PAYMENT_SERVICE_SERVICE_PORT}

This approach is not commonly used as it has one problem: if the client Pod gets created before the service, the service environment variables are unavailable. So you must create the Service before the client Pods come into existence. Otherwise, those client Pods won't have their environment variables populated.

2. Discovery using CoreDNS

Each Kubernetes cluster has a CoreDNS pod which basically acts as a DNS server. It watches the Kubernetes API for new services and creates sets of DNS records for each service.

Now let's see practically how this happens. We can see that the CoreDNS pod is available out of the box by querying, as shown below.

kubectl get pods -n kube-system  

CoreDNS POD

To get data about the CoreDNS service, use the following command.

kubectl get service -n kube-system

When one microservice wants to connect with other microservices, they do a lookup using the CoreDNS service.

To see how this works let's create two simple SpringBoot-based projects.

1. ticket-service (upstream)

2. payment-service (downstream)

Add the following dependencies in pom.xml.

  <dependency>
   <groupId>org.springframework.boot</groupId>
   <artifactId>spring-boot-starter-web</artifactId>
  </dependency>
  <dependency>
   <groupId>org.springframework.cloud</groupId>
   <artifactId>spring-cloud-starter-openfeign</artifactId>
  </dependency>
  <dependency>
   <groupId>org.springframework.cloud</groupId>
   <artifactId>spring-cloud-starter-kubernetes-fabric8-all</artifactId>
  </dependency>

• Step 1: Create the ticket-service application and add the below code:

@SpringBootApplication
@RestController
@EnableFeignClients
@EnableDiscoveryClient
@Slf4j
public class TicketServiceApplication {

  @Autowired
  IPayment iPayment;

  public static void main(String[] args) {
     SpringApplication.run(TicketServiceApplication.class, args);
  }

  @PostMapping("/ticket/pay")
  public String pay(){
     log.info("Calling payment service");
     //wallet payment
     String paymentResponse = iPayment.pay();
     log.info("Payment done..");
     return paymentResponse + ", Ticket confirmed!";
  }
} 

Add IPayment client to the ticket-service project as shown below:

@FeignClient(name="payment-service")
public interface IPayment {

   @PostMapping("/pay")
   public String pay();
}

• Step 2: Create a payment-service application and add below code:

@SpringBootApplication
@RestController
@EnableDiscoveryClient
public class PaymentServiceApplication {

  public static void main(String[] args) {
     SpringApplication.run(PaymentServiceApplication.class, args);
  }

  @GetMapping
  public String status(){
     return "success";
  }

  @PostMapping("/pay")
  public String pay(){
     return "payment successful";
  }
}

• Step 3: Call ticket endpoint using Postman as shown below (which internally calls downstream — payment-service).

Test downstream call works

As we can see, the ticket microservice successfully discovered the payment microservice and got a response from the payment service!!! We can discover and call downstream service now.

Now let's deep dive into internals to see how this works.

First, let's list down and note the IPs of pods and services as shown below.

List IP's

Note : #1 kube-dns IP is : 10.96.0.10 #2 payment-service IP is : 10.108.155.131

Whenever a new POD is created, Kubernetes adds a DNS server entry into the pod's resolv.conf file. (Pod OS — Linux keeps track of networking information inside the /etc/resolv.conf file ). To see this lets sh into one of the pod as shown below.

kubectl exec <PODName> -it sh

Example:

 kubectl exec ticket-service-84479b4b59-rqdjz -it sh

Now navigate to /etc and open resolv.conf as shown below,

# cat /etc/resolv.conf

DNS Name server entry

As we can see in the output, DNS service IP— 10.96.0.10 is available (cross-check with noted IP above #1); hence each POD will be able to contact the DNS service to resolve the service-name to IP address of the actual service to communicate with each other.

As we are on the ticket pod console, let's query the payment-service IP using nslookup as shown below.

payment-service name to IP

As we can see in the above output, the ticket microservice pod can query payment-service (which delegates requests to payment-pods) based on the name, which resolves it to the actual IP where the service is running.

Conclusion

We have successfully deployed a SpringBoot microservice and called a downstream microservice, and saw how Kubernetes discovers services using the CoreDNS server.

Thanks to Federico Kereki for his inputs.