Open sourcing FIAAS
We recently open-sourced FIAAS (backronym for FINN Infrastructure As A Service), our internal lightweight PaaS solution for Kubernetes.
At FINN, we have over 700 microservices running in Kubernetes, as we are in the final stages of a migration process to Kubernetes that started back in 2016. FIAAS is one of the tools that we have used to build a good developer experience when deploying, running and operating applications on Kubernetes. Application developers in FINN are used to delivering changes to production hundreds of times per day, and FIAAS helps us maintain this rate of change.
Design
Kubernetes is a relatively un-opinionated tool, and gives you a lot of flexibility when it comes to running your workloads. It is made to support many different use cases.
FIAAS on the other hand is a highly opinionated tool, which gives you an intentionally limited API, or contract, between your application and the infrastructure it runs on. FIAAS is intended to support a more limited set of use cases. While it will work in many applications, it is optimized for microservices that communicate over the network and do not require persistent state at the application instance level.
FIAAS’s API provides default values for almost all flags to encourage developers to build applications to match the defaults. For example, the default behavior is that applications get a HTTP liveness check at /_/health on port 8080. That is one less decision that application developers need to think about, unless there is a good reason to do it some other way. Just mount the application’s liveness check function at that endpoint, no further configuration needed. By abstracting details away, FIAAS leaves only a minimum of configuration that our developers need to care about for their applications. This means that developers can focus on what is important to them, and not on managing infrastructure. This effectively helps them move faster on what they really want to do, which is work on services that provide value for our users.
Many of the details that are abstracted over can be configured at the cluster or FIAAS instance level. The ones that can’t be configured, can be changed by modifying the FIAAS source code. This creates some room for SRE/operations teams to maneuver and to modify the underlying infrastructure without necessarily involving application developers, since the application configuration in many cases can stay the same.
Application interface
The entrypoint to FIAAS for application developers is a fiaas.yml (the FIAAS config) file in their application’s source repository. The content of this file describes how the application interacts with the infrastructure. Since most settings have a default value, the smallest useful fiaas.yml file an application can have is version: 3. This results in the default values for every option. You can see these default values here to get an impression of how it looks. This is completely usable if the application fits within those constraints. In practice though, most applications will override at least a few of the options. Most applications that do anything useful will need to override resource requirements, replica min/max values and ingress endpoint, since compute resources and ingress hostname/path both are highly dependent on how the application behaves.
If you have experience with Kubernetes you have probably worked with Kubernetes resources as YAML. The size of the Kubernetes resources expressed by a FIAAS config will typically be around 10 times that of the FIAAS config itself.
Deployment
FIAAS uses a Kubernetes custom resource called Application. When the Application resource is created or updated, it will trigger a deployment of the application it refers to. The Application resource looks like this (yes, FIAAS also deploys (parts of) itself):
apiVersion: fiaas.schibsted.io/v1
kind: Application
metadata:
labels:
app: fiaas-deploy-daemon
name: fiaas-deploy-daemon
spec:
application: fiaas-deploy-daemon
image: fiaas/fiaas-deploy-daemon:20190215094822-2ace0ea
config:
version: 3
replicas:
maximum: 1
minimum: 1
# further details omitted for brevityThis resource will typically be updated by a deployment orchestration tool. At FINN we use our own in-house orchestration tool called Pipeline. It currently triggers deployments via a Kafka topic, but is in the process of being transitioned to the CRD model. We have also built integration with Spinnaker using its webhook stage. It is also of course possible to simply use kubectl apply.
When an Application resource is created or updated, the fiaas-deploy-daemon component will read its contents, transform it to native Kubernetes Deployment, Service, Ingress and HorizontalPodAutoscaler resources and create or update these resources as appropriate. This component is designed following the custom controller or operator pattern.
Configuration
Applications will often need some form of configuration. We’re trying to follow the same principle for this as for FIAAS itself. If the application can infer the things it needs to configure from the environment, we’ll try to let it do that. This works for things like e.g. database endpoints, where we often use a headless service with a well-defined name that the application knows. Another configuration mechanism is global configuration and environment variables via FIAAS’s environment specific configuration. Additionally, many applications bundle their configuration for each environment and use the $FIAAS_ENVIRONMENT environment variable which FIAAS always sets to decide which configuration to use. Other applications will use a ConfigMap to inject configuration into the application. If FIAAS sees that there is a ConfigMap with the same name as the application in the namespace it deploys an application, it will mount it and make any applicable key/value pairs in its data section available as environment variables.
Secrets
Some types of configuration are privileged. Database passwords are just one example. FIAAS has a few mechanisms for handling secrets like this. If there is a Kubernetes secret with the same name as the application, that will be mounted and available in the same way as a ConfigMap. We also support using an init container image to pull secrets into an application’s file system via a shared volume. At FINN we use this mechanism to pull secrets from Vault. FIAAS also support pulling secrets from Strongbox, which also uses an init container, but also uses some additional configuration.
FIAAS is not only about Kubernetes though; it also integrates with supporting infrastructure for observability and load balancing.
Metrics
Prometheus is the main system we use to collect application and infrastructure metrics at FINN. By default FIAAS expects that applications expose a Prometheus metrics endpoint at /_/metrics. If the application does this, metrics will be collected and stored in our Prometheus stack. We have generic application metric dashboards that can be used to learn how an application behaves based on the data collected via this mechanism. See an excerpt of this dashboard below:
By following the convention, application developers get a good baseline of observability with very little effort, and can easily use the same mechanism to add their own metrics and dashboards for application and/or business metrics as well.
FIAAS also supports collecting and aggregating metrics to Datadog.
Logs
There are several approaches to handling log aggregation for applications running in Kubernetes. In FIAAS, applications are expected to log to /dev/stdout. This contract allows the team that owns the cluster that applications run in to decide where the logs should be aggregated. In FINN we have a fluentd setup that sends all application log streams to our Elasticsearch + Kibana setup, where application developers can search their aggregated logs.
Traffic ingress
Microservices like to talk to each other. Many of our applications talk HTTP, and many also need to respond to requests from the internet. FIAAS’s default behavior is that you have a root domain suffix e.g. services.example.com, and that applications will be available at $application_name.services.example.com. If an application needs to be available on a different domain or path, this can be configured in its FIAAS config.
Lifecycle
Just running and deploying applications is not enough. With hundreds of applications, applications will be decommissioned from time to time. FIAAS supports this use case too; by deleting the Application resource for an application, all other resources managed by FIAAS will also be deleted.
Open source
FIAAS is open source. Creating the remote repositories on github.com and git push-ing the code there was the easy part. We have adapted our process into something that can work in an open environment. This meant changing and formalizing processes that used to be institutional knowledge as our governance model and code of conduct. Additionally we had to migrate our CI/CD tooling and processes from our internal systems to tools that fit better an open source world. We have prioritized these parts in addition to the code because we think it is important to build a project that can sustainably continue in an open source manner.
FIAAS works very well for us at FINN, and for many other companies in the Schibsted family, but there is still work to be done. By open sourcing it, we hope to help others get up and running with Kubernetes, but also to help us adapt and refine FIAAS to make it a better product for everyone.
Read more at https://fiaas.github.io or join us in our Slack channel #fiaas on https://kubernetes.slack.com.
If you are interested in learning more about the rest of our infrastructure, you should read this post:
