Overcoming Configuration Complexity: Streamline Your Workflow with DevX by Stakpak

Hanyelfouly
9 min readMay 3, 2023

A single configuration tool to generate, share, and validate ALL your workflows and infrastructure code.

In the world of software development, configuration complexity can be a significant roadblock, leading to a loss of productivity, increased technical debt, and a higher likelihood of errors. As projects grow in size and scope, managing configurations across multiple environments can become a nightmare. However, by understanding the challenges and implementing best practices, developers can streamline their workflow and avoid the pitfalls of complex configurations. In this article, we will explore the challenges faced by developers due to configuration complexity, discuss best practices for simplifying configuration management, and introduce DevX — a powerful solution that can help you overcome the configuration complexity nightmare.

So, if you’re tired of struggling with complex configurations and are ready to boost your productivity, this article is a must-read for you!

According to GitLab 2022 Global DevSecOps Survey:

  • 44% of devs use 2–5 tools
  • 41% of devs use 6–10 tools
  • 69% want to consolidate their tool-chains
  • 40% of devs spend 25–50% on integrating & maintaining tool-chains
  • 33% of devs spend 50–100% on integrating & maintaining tool-chains

The devs aren’t happy with all of the context-switching. Other concerns included slowed development velocity, cost increases, and difficulty in retaining developers. Clearly, teams are tired of paying the “toolchain tax.”

The CNCF landscape is ever-growing

Source: CNCF Native Interactive Landscape

Challenges of Configuration Complexity

Configuration complexity arises from several factors, including:

  1. Scale: As the size of an infrastructure grows, the number of components and the complexity of their interactions increases, making it more difficult to manage and maintain.
  2. Diversity: When multiple different types of infrastructure components are used, such as different operating systems, hardware platforms, and software applications, it can be more difficult to ensure that they are all properly configured and work together seamlessly.
  3. Security: Ensuring the security of an infrastructure is a critical task, but it can also make configurations more complex. This is because security measures often require multiple layers of protection, including firewalls, access controls, and encryption, which can increase the complexity of the configuration process.
  4. Versioning and updates: Technology is constantly evolving, and new versions of infrastructure software are released frequently. Keeping track of configuration settings across multiple versions and ensuring that they are updated correctly is a major challenge.
  5. Automation: While automation can help to simplify infrastructure management in some ways, it can also add to the complexity of configurations. This is because automation tools often require specialized knowledge and configuration skills, and can also introduce additional dependencies and potential points of failure into the infrastructure.

Best Practices for Simplifying Configuration Management

To overcome the challenges posed by configuration complexity, consider the following best practices:

  1. Centralize configuration management: Use a single source of truth for all your configurations, ensuring consistency and reducing the likelihood of errors.
  2. Automate: Leverage automation and code generation tools to generate and validate configurations consistently across environments, and to detect and remediate misconfiguration before applying. To avoid the common pitfall of applying then detecting misconfiguration.
  3. Version control and tagging: Track configuration changes using version control systems to identify issues and maintain a history of changes with the ability to easily roll back in case of failure.
  4. Standardize: Adopt standard conventions and practices for configuration management to reduce complexity and improve maintainability.

What is DevX:

DevX is a configuration management tool for building developer-centric interfaces for your internal developer platform (IDP).

Use DevX to build flexible internal standards, prevent misconfigurations early, and enable infrastructure self-service.

source: DevX documentation

Developer-centric configurations using a single config for all environments, for all vendors!

Configuration language

We use CUE to write strongly typed configurations. You can now shift YAML typos left, instead of detecting errors after applying configurations.

CUE is the result of years of experience writing configuration languages at Google, and seeks to improve the developer experience while avoiding some nasty pitfalls. CUE looks like JSON, while making declarative data definition, generation, and validation a breeze. You can find a primer on CUE here.

Definitions

Some useful definitions and explanations to get an understanding of our approach. If you find it overwhleming you can get started quickly using the next section and get a deeper understanding as you go along.

Project: a collection of devx configuration files in a git repository. Projects have unique module identifiers like devopzilla.com/user-service.

Stack: a set of components and is how you define your workloads and its dependencies. A stack is a contract between developers and platform designers.

Component: represents a workload or a resource. A component has an id, a set of traits, labels, and fields. A component can be low level (e.g. RDS instance, Helm chart, Kubernetes resource) or as high level as you want (e.g. Django app, Postgres database, S3 API compatible bucket).

Trait: represents a capability of a component and associated fields. You add a trait to a component to make use of the platform’s capabilities, or define your own. Traits add fields to the component to allow it to be configure either by the developer, the platform team, or automatically by DevX (e.g. pull data from the environment or auto-generate secrets).

Transformer: a data transformation function. A transformer is used to enrich components with more data until it’s ready to deploy using various drivers. Transformers are used to encode your platform’s best-practices and turn the abstract data defined by the stack into a form existing tools can operate on.

Resources: the output of devx that is applied to your favorite platform (e.g. Kubernetes manifests, Terraform code, GitLab pipelines).

Driver: used to write resources as configuration files to be consumed by existing tools like Kubectl, Terraform, ArgoCD, and Github Workflows.

Prerequisites

  • Install DevX (Option 1: Homebrew)
brew tap devopzilla/guku-devx
brew install guku-devx

After you install DevX

  1. Head to the getting started page on the documentation to get started and understand the basics.
  2. Get familiar with the basics through our quick intro.
  3. Understand the concepts and guiding principles such as declarative Traits and Transformers.
  4. Check out DevX playground to start experimenting.

Now let’s take a very popular framework like the famous python Django which is used by companies all over the world and apply the concepts and approach of devx to it.

Project directory structure:

cue.mod
└── module.cue
stack.cue
.gitignore
-- remaining app code --

Define infrastructure abstractions: DevX allows you to create your own abstractions, which can help you tailor your configuration management to your specific needs.

"django-app": {
traits.#Workload
containers: {
default: {
image: "myapp/myapp"
}
}

traits.#Exposable
endpoints: {
default: {
ports: [{
port: 8000
health: {
path: "/api/health"
periodSeconds: 60
}
}]
}
}
}

Shift-left validation: DevX enables early detection of misconfigurations by treating configurations as strongly typed and immutable data, ensuring that errors are caught before they are pushed to production. Everything defined with DevX is strongly typed. Validate configurations while writing. Error examples:

  • Missing required field
  • Wrong field type
  • Conflicting configurations
"django-app": {
traits.#Workload
containers: {
default: {
image: string
}
}

traits.#Exposable
endpoints: {
default: {
ports: [{
port: uint & >1024 & <65536
health: {
path: string
periodSeconds: uint & <300
}
}]
}
}
}

Achieve parity across environments: DevX encourages workload definitions to be as similar as possible across environments, pushing implementation details to the platform layer.

If you prefer video format check this out

When we run this command: devx build Compose
We get the following as docker compose output:

version: "3"
volumes: {}
services:
django-app:
image: myapp/myapp
environment: {}
depends_on: []
ports:
- "8000:8000"
command: []
restart: always
volumes: []

When we run this command: devx build Kubernetes
we get the following as Kubernetes manifest output:

kind: Deployment
apiVersion: apps/v1
metadata:
name: django-app
labels:
app: django-app
spec:
selector:
matchLabels:
app: django-app
template:
metadata:
labels:
app: django-app
annotations: {}
spec:
containers:
- name: default
image: myapp/myapp
command: []
args: []
env: []
restartPolicy: Always
serviceAccountName: django-app
securityContext:
runAsUser: 10000
runAsGroup: 10000
fsGroup: 10000
---
kind: ServiceAccount
apiVersion: v1
metadata:
name: django-app
labels:
app: django-app
---
kind: Service
apiVersion: v1
metadata:
name: django-app
spec:
ports:
- name: "8000"
port: 8000
selector:
app: django-app
type: ClusterIP

When we run this devx build ecs command
We get the following output as Terraform config in json for ecs:

{
"data": {
"aws_service_discovery_dns_namespace": {
"ecs_cluser-XXXXXXX": {
"name": "cluser-XXXXXXX.ecs.local",
"type": "DNS_PRIVATE"
}
},
"aws_vpc": {
"prod": {
"tags": {
"Name": "prod"
}
}
},
"aws_ecs_cluster": {
"cluser-XXXXXXX": {
"cluster_name": "cluser-XXXXXXX"
}
},
"aws_subnets": {
"prod": {
"filter": [
{
"name": "vpc-id",
"values": [
"${data.aws_vpc.prod.id}"
]
},
{
"name": "mapPublicIpOnLaunch",
"values": [
"false"
]
}
]
}
},
"aws_kms_alias": {
"cluser-XXXXXXX": {
"name": "alias/ecs/cluser-XXXXXXX"
}
},
"aws_cloudwatch_log_group": {
"cluser-XXXXXXX": {
"name": "/aws/ecs/cluser-XXXXXXX"
}
},
"aws_subnet": {
"prod": {
"count": "${length(data.aws_subnets.prod.ids)}",
"id": "${tolist(data.aws_subnets.prod.ids)[count.index]}"
}
}
},
"resource": {
"aws_security_group": {
"ecs_service_internal_django-app": {
"name": "ecs-service-internal-django-app",
"vpc_id": "${data.aws_vpc.prod.id}",
"ingress": [
{
"from_port": 0,
"to_port": 0,
"protocol": "-1",
"cidr_blocks": "${data.aws_subnet.prod.*.cidr_block}",
"description": null,
"ipv6_cidr_blocks": null,
"prefix_list_ids": null,
"self": null,
"security_groups": null
}
],
"egress": [
{
"from_port": 0,
"to_port": 0,
"protocol": "-1",
"cidr_blocks": [
"0.0.0.0/0"
],
"description": null,
"ipv6_cidr_blocks": null,
"prefix_list_ids": null,
"self": null,
"security_groups": null
}
]
}
},
"aws_service_discovery_service": {
"django-app": {
"name": "django-app",
"dns_config": {
"namespace_id": "${data.aws_service_discovery_dns_namespace.ecs_cluser-XXXXXXX.id}",
"dns_records": {
"ttl": 5,
"type": "A"
},
"routing_policy": "MULTIVALUE"
},
"health_check_custom_config": {
"failure_threshold": 5
}
}
},
"aws_iam_role": {
"task_django-app": {
"name": "task-cluser-XXXXXXX-django-app",
"assume_role_policy": "{\"Version\":\"2012-10-17\",\"Statement\":[{\"Sid\":\"ECSTask\",\"Effect\":\"Allow\",\"Principal\":{\"Service\":\"ecs-tasks.amazonaws.com\"},\"Action\":\"sts:AssumeRole\"}]}"
},
"task_execution_django-app": {
"name": "task-execution-cluser-XXXXXXX-django-app",
"assume_role_policy": "{\"Version\":\"2012-10-17\",\"Statement\":[{\"Sid\":\"ECSTask\",\"Effect\":\"Allow\",\"Principal\":{\"Service\":\"ecs-tasks.amazonaws.com\"},\"Action\":\"sts:AssumeRole\"}]}"
}
},
"aws_iam_role_policy": {
"task_django-app_default": {
"name": "task-cluser-XXXXXXX-django-app-default",
"role": "${aws_iam_role.task_django-app.name}",
"policy": "{\"Version\":\"2012-10-17\",\"Statement\":[{\"Sid\":\"SSMExec\",\"Effect\":\"Allow\",\"Action\":[\"ssmmessages:CreateControlChannel\",\"ssmmessages:CreateDataChannel\",\"ssmmessages:OpenControlChannel\",\"ssmmessages:OpenDataChannel\"],\"Resource\":\"*\"},{\"Effect\":\"Allow\",\"Action\":[\"logs:DescribeLogGroups\"],\"Resource\":\"*\"},{\"Effect\":\"Allow\",\"Action\":[\"logs:CreateLogStream\",\"logs:DescribeLogStreams\",\"logs:PutLogEvents\"],\"Resource\":\"${data.aws_cloudwatch_log_group.cluser-XXXXXXX.arn}\"},{\"Sid\":\"SSMDecrypt\",\"Effect\":\"Allow\",\"Action\":[\"kms:Decrypt\"],\"Resource\":\"${data.aws_kms_alias.cluser-XXXXXXX.target_key_arn}\"}]}"
},
"task_execution_django-app_default": {
"name": "task-execution-cluser-XXXXXXX-django-app-default",
"role": "${aws_iam_role.task_execution_django-app.name}",
"policy": "{\"Version\":\"2012-10-17\",\"Statement\":[{\"Sid\":\"ECSTaskDefault\",\"Effect\":\"Allow\",\"Action\":[\"ecr:GetAuthorizationToken\",\"ecr:BatchCheckLayerAvailability\",\"ecr:GetDownloadUrlForLayer\",\"ecr:BatchGetImage\",\"logs:CreateLogStream\",\"logs:PutLogEvents\"],\"Resource\":\"*\"},{\"Sid\":\"ECSTaskSecret\",\"Effect\":\"Allow\",\"Action\":[\"ssm:GetParameters\",\"secretsmanager:GetSecretValue\",\"kms:Decrypt\"],\"Resource\":[\"arn:aws:kms:us-west-1:XXXXXXX:key/*\"]}]}"
}
},
"aws_ecs_service": {
"django-app": {
"name": "django-app",
"cluster": "${data.aws_ecs_cluster.cluser-XXXXXXX.id}",
"task_definition": "${aws_ecs_task_definition.django-app.arn}",
"desired_count": 1,
"launch_type": "FARGATE",
"wait_for_steady_state": true,
"enable_execute_command": true,
"service_registries": {
"registry_arn": "${aws_service_discovery_service.django-app.arn}"
},
"network_configuration": {
"security_groups": [
"${aws_security_group.ecs_service_internal_django-app.id}"
],
"subnets": "${data.aws_subnets.prod.ids}"
}
}
},
"aws_ecs_task_definition": {
"django-app": {
"family": "cluser-XXXXXXX-django-app",
"network_mode": "awsvpc",
"execution_role_arn": "${aws_iam_role.task_execution_django-app.arn}",
"requires_compatibilities": [
"FARGATE"
],
"cpu": "256",
"memory": "512",
"task_role_arn": "${aws_iam_role.task_django-app.arn}",
"container_definitions": "[{\"essential\":true,\"name\":\"default\",\"image\":\"myapp/myapp\",\"cpu\":256,\"memoryReservation\":512,\"portMappings\":[{\"containerPort\":8000}],\"linuxParameters\":{\"initProcessEnabled\":true},\"environment\":[],\"secrets\":[],\"healthCheck\":{\"command\":[\"CMD-SHELL\",\"exit 0\"]},\"logConfiguration\":{\"logDriver\":\"awslogs\",\"options\":{\"awslogs-group\":\"/aws/ecs/cluser-XXXXXXX\",\"awslogs-region\":\"us-west-1\",\"awslogs-stream-prefix\":\"django-app\"}}}]"
}
}
}
}

Configuration as Data: By treating configurations as pure data, DevX simplifies infrastructure as code (IaC), making it easier to onboard developers and debug misconfigurations at scale.

Leverage Extensibility: DevX’s extensibility features enable you to define custom Traits and Transformers to support unique use cases and tailor the tool to your specific needs.

// Define a trait to be used by end users
#DjangoApp: v1.#Trait & {
traits.#Workload
containers: default: {
image: string
command: ["/app/entrypoint.sh"]
}
traits.#Exposable
endpoints: default: ports: [
{
port: 8080
target: 80
},
]
allowedHosts: [...string]
}

// Tell devx what to do when it encounters the trait
#DjangoAppTransformer: v1.#Transformer & {
#DjangoApp
allowedHosts: _
endpoints: _
containers:
default:
env:
ALLOWED_HOSTS: strings.Join([
"localhost",
endpoints.default.host,
for _, host in allowedHosts {host},
], ",")
}

// Devs use the new trait
"django-app": {
#DjangoApp
containers: default: image: "myapp"
allowedHosts: ["myapp.stakpak.dev"]
}

Advantages of DevX Over Traditional Approaches:

  1. Strongly Typed Configurations: By treating configurations as strongly typed data, DevX helps catch configuration errors early, reducing the risk of failures in production.
  2. Simplified Infrastructure as Code: DevX simplifies IaC by treating configurations as pure data, making it easier for developers to understand, debug, and maintain their configurations.
  3. Custom Abstractions: DevX allows you to create your own abstractions, giving you the flexibility to adapt the tool to your organization’s unique requirements.
  4. Shift-Left Validation: By detecting misconfigurations early in the development process, DevX helps reduce the time and effort spent debugging configuration issues. Shifting left allows teams to create conventions and streamline processes while implementing guardrails and upholding the agreed upon standards.

Stakpak — we are on a mission, to empower developers to build the best products with joy.

Enabling engineering teams to write portable configurations that allow them to evolve their platforms without disrupting delivery speed. Configurations packaged with Stakpaks are easy to re-use, share, and validate very early on in the SDLC.

We build open source and commercial tools to help you → Develop faster, Deploy anywhere.

Enable developers to build their own Heroku.

We want to build truly technology-agnostic configurations, where developers define their intents and Stakpak figures out how to make things happen on any cloud provider, and underlying platforms. Where you can have your own Heroku-like experience on your cloud-provider and technology platform of choice.

Use developers’ intent to build all the configurations needed from development to production. For any technology stack, and any provider.

Got your interest!

If you would like to know more about how Stakpak hub or DevX reach out to us at contact@stakpak.dev

We also provide Platform engineering and DevSecOps consulting services.

Don't forget to ⭐️ our repo on GitHub.

References

--

--

Hanyelfouly

A metalearner with a passion for interaction between humans and technology. Focused on developer experience, wellbeing and peak performance