How to program infrastructure with Pulumi part 2

Previously we covered high level Pulumi concepts and compared it to Terraform. Today let’s dwell a bit more on the available Pulumi features like default service backend, state and secrets management or available Terraform integrations.

Default service backend

CLI is tightly coupled to the default service backend. Once you are logged in, each project you create will be showed in the web dashboard. The checkpoint files (infrastructure state) will be also kept there. That’s a bold decision taking into consideration that checkpoints contains secrets and passwords to your cloud services.

Default service backend basic usage is free of charge, however, its offer is tiered, and depending on features you want, you might need to upgrade to paid version.

In the free Community Edition version dashboard you can review the projects, check the activity log and list the available resources in your stacks grouped by categories like Containers, VMs, SQL, Queues and etc.

Notable features in paid versions are:

  • CI/CD integration
  • Basic or advanced secrets management
  • Self hosting the https://app.pulumi.com
  • Automatic state locking
  • Deployment history (supports rolling back to previous infrastructure state)

Available tiers and features can be checked here.

State management

As described above, by default, checkpoint files are stored in the default service backend. For security reasons this will be a big no for many companies. These files may contain passwords and keys, thus, it is not recommended to host them at a third party SaaS provider.

However, Pulumi supports third party backends like S3 or GCS. You can store checkpoints, for example, in your’s company AWS account where you are the one controlling access policies.

Switching state backend to S3 bucket is very easy and will be presented in next installment. Unfortunately, as of today, self-managed backends do not support state locking. If you do not plan to buy paid Pulumi subscription that includes locking, you will have to implement it by yourself.

It shouldn’t be a big problem — your language of choice probably already has a decent locking library, e.g., Python’s python-dynamodb-lock.

Stack configuration

  • Storing the secrets provider configuration (explained in next section),
  • Storing stack variables

To store an variable in configuration file you should use pulumi config setcommand. Note that variable is also prefixed by a stack name.

$ pulumi config set examplevar "examplevalue"
$ grep examplevar Pulumi.dev.yaml
mon:examplevar: examplevalue

How to reference these variables in program depends on language you use. Python example:

import pulumi
cfg = pulumi.Config()
val = cfg.require('examplevar')
val2 = cfg.get('examplevar')

Difference between cfg.require() and cfg.get() is that in case of absence of the variable the former will raise an exception and the latter will return none.

Secrets management

For example:

pulumi config set --secret influxdb_admin_password S3CR3T_PA55W0RD

Encrypted secret will be stored alongside stack variables. This is an imperfect solution — you end up with secret hashes in VCS —they are hashes, but still, having them in VCS is considered a bad practice.

However, Pulumi uses general purpose languages to manage infrastructure; you can easily obtain your passwords from AWS SSM or Azure Vault and avoid using the built-in secrets management function. You can even use native Pulumi primitives for that:

somepassword = pulumi_aws.ssm.get_parameter(with_decryption=True, name="/some/password_in_ssm")

As said before, default encryption provider is https://app.pulumi.com. You can change it to an alternate provider. My personal opinion is that it is wiser to choose external encryption provider like, e.g., KMS in your AWS account. Not only you control the key revocation process, but you also have access to whitepapers describing how encryption is done, and how high entropy is achieved. Currently supported encryption providers are:

  • AWS KMS
  • HashiCorp Vault
  • Azure Vault
  • GCP KMS

Encryption provider cannot be changed for already created stack(or at least there is no documentation on that). You must specify encryption provider when creating a stack:

pulumi new aws-python --dir mon --secrets-provider="awskms://KMS_KEY_ID?region=eu-west-1"

Terraform integrations

To support soft transition from Terraform to Pulumi a pulumi-terraform package was created(in description it is also referred to as bridge so it is a bit confusing). You can use it to reference infrastructure created by Terraform in Pulumi programs. For example, using Pulumi you may deploy an application in a VPC referenced from a Terraform state.

In next installment we will use the knowledge acquired so far to create an real infrastructure using Pulumi! Stay tuned.

Feel free to contact me at ks@kscloud.pl. If you want to receive my posts over your email subscribe to my substack at https://cloudtalks.substack.com/.

https://kscloud.pl