Service Registration and Discovery & Configuration Management

About Me: Application Architect at Oildex, a Services of Transzap Inc,

Scaling of distributed applications and infrastructure is very important in Microservices Architecture in an automated way. Without proper automation a team just continuously struggles and wastes time behind silly things like application configuration, gateway/load balancer configuration, automation tools, etc. Service Registration and Discovery & Configuration Management have become very crucial to keep architecture clean and also for the team to focus on a core part of development rather than maintenance and resolving fires.

Current Picture/What Problem are we solving?

  • Routing traffic to the correct service (Hardcoded IP address approach & DNS Lookup)
  • Manual Intervention of Adding and Removing new services to/from the pool
  • Partnership with Third party consulting firms, while credentials are not secured.
  • Unraveling Configuration Management (Puppet, Chef, ..) issues every time you touch
  • Hassle of managing static Application and Nginx configuration files
  • Long live one time generated API Keys
  • Microservices Scalability
  • Service Failures
Overall Microservices Architecture looks like this without Service Registry & Discovery

What is Service Registry & Discovery?

Service Registration — The process of a service registering its location in a central registry. It usually registers its host and port and sometimes authentication credentials, protocols, versions numbers, and/or environment details.
source: Open-Source Service Discovery by Jason Wilder
Service Discovery — The process of a client application querying the central registry to learn of the location of services.
source: Open-Source Service Discovery by Jason Wilder

Benefits

  • Services can be started and stopped in any order
  • Servers may fail/be restarted
  • Better Scalability for TRUE distributed system
  • Consistent view of Reactive Configuration
  • Consistent view of services

Service Registry & Discovery Tools

Consul = Zoo Keeper + Nagios + DNSMasq + Scriptings + …

Consul brings all in one package

What Does do Consul offer?

  • Service Registration and Discovery
  • Health Checks
  • Dynamic Configuration — KV Store
  • Security:
  1. TLS
  2. Access Control List (ACL) — Fine Grained access to data & API
  • Key Generation
  • Dynamic Load Balancing
  • Support for Single & Multi Data Centers
  • Consul Template
  • Data Encryption
  • Integrates well with Containers
  • Simple REST API
  • CLI — Command Line Interface
  • Web-based Interface for administration
  • JSON
  • Support for any programming language
  • Gossip Protocol

Consul Dependency: Go

Available REST endpoints for?

  • kv — key value
  • agent — API for dealing with an agent
  • catalog — dealing with datacenter catalog
  • health — service health checks
  • sessions — group operations and manage consistent view
  • events — fire fast gossip based events
  • acl — setup access control lists and security for Consul
  • status — check the status

Consul Enterprise Features

  • Automated Backups
  • Automated Upgrades
  • Enhanced Read scalability
  • Advanced network models
  • Redundancy Zones



Desired Future Picture

  • Dynamic Configuration — Consul
  • Service Registry & Discovery — Consul
  • Credential Store — Vault
  • API Key Generation — Consul

Dynamic Configuration

Application configuration has always been a challenge in the organization without DevOps mindset. Which also opens up a risk of potential security issues when all kinds of credentials for different organizations get stored into source control in form of plain text. Also, companies have been working with third-party consultants at large scale without any hesitations.

An Ideal approach of Application Configuration is:

  • Key-Value storage tool with ability to encrypt data
  • Key-Value storage tool with availability of HTTP REST API
  • Key-Value storage tool with availability of CLI
  • Key-Value storage tool with availability of Role-based access for organization
  • Key-Value storage tool with audit control
  • Microservices should be able to pull configuration using HTTP REST API for specific application and environment at the startup without any extra dependencies without Container Approach.
  • Ability to Reload Configuration using specific REST endpoint in application context without Bouncing API.

KV Subcommands:

  • delete: Removes data from the KV store
  • export: Exports a tree from the KV store as JSON
consul kv export prefix > cat DirectoryPath/FileName.json
  • get: Retrieves or lists data from the KV store
  • import: Imports a tree stored as JSON to the KV store
consul kv import @DirectoryPath/FileName.json
  • put: Sets or updates data in the KV store


I hope this post has helped you. If you enjoyed this article, please don’t forget to clap👏 ! I would love to know what you think and would appreciate your thoughts on this topic. You can also follow me on Medium, GitHub and Twitter for more updates.