Introducing discfg, a Configuration Service for Your Applications
Distributed and Serverless
An application configuration service is used to maintain a configuration in a consistent or conventional fashion. It helps separate concerns in software architecture. The service allows for configuration changes to be updated from outside the application, though the service often requires hosting itself.
Sometimes this service is built right into the application where configuration data is stored alongside other application data (look at applications such as WordPress for example).
It’s a common need among many applications, but in particular it’s a larger concern for distributed systems or microservices. When you move away from the monolithic application, you begin to need configuration information in many places so a service makes sense for maintainability reasons alone.
Though there are actually many reasons you’d want a dedicated configuration service. Here are just a few reasons:
- You don’t want to store sensitive credentials in your application code where it might be accidentally committed into revision control.
- You don’t want to store config data in the application database.
- You want to easily share configurations with other trusted developers.
- You want to manage customer’s application configurations without them being able to.
- Any other configuration access control scenario.
- You want a consistent way to manage configuration and update values on the fly.
- You’re building stateless microservices where you can’t easily keep a configuration around (ie. AWS Lambda).
Why Did I Build it?
This simple tool came about from a need of mine when building microservices with AWS Lambda. One of the downsides to Lambda is that you can’t easily set environment variables. With ECS, which uses Docker, you sure can and I found myself gravitating more and more toward ECS despite the better fit with Lambda.
Why was that? Well, it was because configuration and state was a real pain with AWS Lambda. Nothing persists there (unlike Azure Functions) so you have to bake everything into the function code. That’s not maintainable. A configuration service is clearly needed (and I’m surprised AWS has not offered one yet).
I wasn’t so interested in service discovery at the time, just configuration. Specifically a simple, yet conventional, key/value store.
So I looked into etcd and some other solutions…Before I knew it, I was back to worrying about the health of clusters — for simple configuration needs. Setting aside the fact that etcd was costing more to host than the actual serverless application itself, it was simply a maintenance I didn’t want.
Now don’t get me wrong. I love etcd. I drew so much inspiration from it and discfg is simply not a replacement for etcd (nor was that ever the intent). It is, however, the configuration tool for many of my needs (maybe yours too). Even still, I’ll usually use it as a point of reference.
Since the whole serverless concept is fairly new, there really aren’t any solutions around like this. It really just required a bit of boilerplate code in each of your functions every time you turned around.
Discfg is intended to be hosted in AWS and be serverless. It will have no problem scaling and there is no major concern with monitoring either. It just sits there and does its job.
How Does it Work?
Simple. It’s just a key/value store, but it’s the conveniences around that key/value store that make it what it is. It can expose a RESTful API to make accessing it easy from nearly any web application.
discfg is essentially a convenient wrapper around DynamoDB (with more storage engines to come)
It’s written in Go so, if your web application is written in Go, you could even just import the project to use a consistent API for config values and never need to run the API server. You would need to configure AWS credentials of course, but that should be fairly straightforward.
Most folks will be using AWS Lambda + API Gateway to host the API. Terraform and Apex were used to build and deploy this, so you’d need those tools to get it setup. It should just be a few simple commands after that (assuming you have an AWS account and set credentials on your system, most likely because you used the aws-cli tool before).
> apex infra apply -var 'aws_account_id=XXXXX'
> apex deployAfter that, you can simply access the RESTful API to create configurations, set, read, update, and delete keys.
The discfg binary is a CLI for your local machine. So you can work with configurations from your command line. Need to update something? It’s as quick as running:
> discfg set mycfg mykey "some value"You can set the current configuration that you’re using so you only need to specify the keys and values when working with the CLI. It’s just a simple shortcut for convenience.
> discfg use mycfg
> discfg set mykey "some value"Retrieving the value? Easy:
> discfg get mycfg mykeyYou can even set data from a file:
> discfg use mycfg
> discfg set mykey -d file.jsonDepending on how your application refreshes configuration data, you could very easily update its configuration straight from your command line in seconds.
Wait a Minute…What About Security?
Like other configuration tools, it does not take into consideration security itself.
However, keep in mind that through Amazon you have access control with IAM. No one is updating this configuration except for those you allow. Remember that API Gateway allows the use of authentication tokens as well. So you’re really covered in terms of security.
How Can my Apps Use it?
They can use the API which can be restricted to certain IP addresses or protected with an access token. Depending on your application, you may even want to cache the configuration data so you aren’t retrieving it more than necessary.
You can get a bit more sophisticated as well. For example, you could adjust the API so that only certain operations are available — for instance just read operations. This way your front-end applications can get the settings they need without allowing any curious people to inspect your endpoint to then add their own data.
You could then update data from the CLI or a separate API endpoint that has more permission. AWS API Gateway offers you a lot of flexibility and options here (a lot is an understatement).
Alternatively, if your application is built with Go, you could import discfg into your project and call its commands from Go.
Trade-Offs? Other Tools?
Well, it’s not really about competing with other tools for performance.
DynamoDB is pretty darn fast, but it’s no Raft. DynamoDB is technically an eventually consistent storage engine. But it does have conditional writes and discfg can leverage this feature. So you can work with some guarantees when necessary.
There’s most definitely a huge difference between something like the locking etcd system, the Raft consensus and then DynamoDB. Huge.
What is your application doing and what are you trying to build? If you’re looking to build systems level services then I would probably take a look at etcd or something else. If you’re trying to configure a web application, then I really see no reason not to use discfg…But that’s going to depend on the needs of your application.
Service discovery, for instance, may be better handled by other tools.
One big trade-off is watching for updates. With etcd you can have your application watch for a key value to update. Since discfg was designed to run using AWS Lambda, there can’t be long polling. So this feature doesn’t exist today (there’s some ideas around using AWS Kinesis to work in similar functionality in the future).
This is part of why I don’t really compare discfg to things like etcd, Zookeeper or Consul. It’s just a different use case.
Any other comparisons? Yes, just one comes to mind. Webdis is an HTTP interface for Redis. It’s fairly similar but again you’re on the hook for hosting. Though Webdis does make it a point to address access control itself unlike many other tools.
Redis is yet again an altogether different storage engine too. It comes with slightly different use cases, but nothing says you can’t use discfg for a small fast cache too. Who knows, maybe discfg will even use Redis as storage engine one day.
I suspect the simplicity of discfg will lead it to being used in a variety of ways. I think it makes DynamoDB a bit more approachable for starters.
It’s Free as in Beer and it’s Also in Beta
It’s a free tool with an Apache 2.0 license. Keep in mind this tool is in beta. Definitely, please, bring up bugs on GitHub if you spot any.
In the future, I plan to add support for S3 as a storage engine too. There might be reasons why one would choose S3 over DynamoDB. It all depends.
Feedback is very much welcome too. There’s some issues and a loose roadmap for the tool on the GitHub project.
