Below is my analysis, that covers the difference between AWS Savings Plan and Reserved Instances based on atttibutes like “pricing”, “Instance”, “account” and other attributes.

Note: As AWS keeps on enhancing the Savings plans/Reserved instances features, this analysis is performed in Feb 2024. Always, refer to the latest information in official documentation shared by AWS. (Read here about savings plans and here about Reserved instances.)

Pricing Attributes:

I tried to name the parameters so that they are self-explanatory. However, I will still explain some of them to make it more understandable.

Below is the comparison table for Pricing attributes.

Commitment Type: When you buy the savings plan, you commit to AWS that you will make x dollar ($) of usage per hour for 1 OR 3 years. However, when you purchase Reserved instances, you commit AWS that you will buy x number of Ec2 instances for 1 OR 3 years. So the difference is, in the earlier, commitment is on dollars per hour, and for later it is number of Ec2 instances.

Instance Attributes:

As you see below, the Savings plan provides more flexibility than Reserved instances when it comes to instance-specific attributes.

Fixed: denotes that you cannot change once a purchase is made.

Flexible: denotes that you can even make modifications after the purchase.

Account Attributes:

If you have a large number of AWS accounts as part of AWS Organizations, then a savings plan is the only option if you want to make commitment at the Organization level rather than the AWS account level.

Other Attributes:

The explanation for some parameters falling under “Other attributes” is as follows:

Capacity Reservation: this means, if AWS will reserve the capacity for your commitment. AWS only reserves the capacity for ZONAL Reserved Instances.

Selling/Buying On MarketPlace: You can sell the Reserved instances if you over-committed in the past, and now your Reserved instances have become unusable because of a change in business needs. The same is the case with buying at the marketplace.

Automatically Apply to Lambda/Fargate: Only applicable for a savings plan. If you commit for x dollars hourly commitment, you do not need to explicitly mention if your usage will be applicable for ec2 or lambda or fargate. It will automatically provide you the % savings on all the services it supports.

Updating Plans after purchase: If you have any flexibility to modify your commitment. It’s only possible conditionally with convertible reserved instances.

Queuing a purchase: If your existing commitment is nearing the expiration date, you can already make a purchase for future commitment starting from the date in future. Refer the table above to check for possibilities.

Supported AWS Services: Not all the services are eligible for savings plan OR reserved instances. Check the table above for eligible AWS services falling in each category.

Conclusion:

1. AWS Savings plan is more flexible than reserved instances, but your decision depends on whether you want more savings or more flexibility.
2. AWS recommends using Savings plans over reserved instances because of flexibility. Assess your use case and make the commitment carefully and optimize your cloud spending accordingly.

If you liked this article, please show your appreciation by clapping 👏 below! Happy Learning!

However, during ReInvent-2023, AWS released another feature known as EKS Pod Identity which simplifies this requirement. Let’s see how.

Before this, IRSA(IAM roles for Service Accounts) was a commonly used approach. You can read more about IRSA here.

Limitations With IRSA:

1. You need to manage the OIDC provider. In some cases, OIDC is managed by other teams, and hence the control is compromised for the infra team.
2. You need to mention the OIDC provider in the IAM trust policy, which adds complexity.
3. You need to mention IAM role ARN using annotations in ServiceAccount.
4. One IAM role is restricted to an EKS cluster. Cannot be used across clusters. So need to create/manage multiple IAM roles for the same purpose.
5. If you need to change the IAM role associated with a service account, you may need to recreate the pods using that service account to pick up the change.

Benefits of EKS Pod Identity:

1. IAM roles can be used across multiple EKS clusters. Need to create and manage a lesser number of IAM roles.
2. IAM role trust policy can have a generic simplified assume role principal pods.eks.amazonaws.com.
3. Using CreatePodIdentityAssociation API, an association is made between this IAM role and a Kubernetes service account. So a generic IAM role and generic service accounts are created and associated with each other, without any need to use the IAM role arn in the service accounts.
4. It works based on the agent running as a Daemon in each node. Just another pod supporting to help grant permissions to pods.
5. Supports Role Session Tags: This allows for creating more sophisticated permission policies that are reusable and can match specific attributes like cluster names, service accounts, or namespaces.
6. No Additional Cost: Amazon EKS Pod Identity is available at no extra charge, which can be particularly beneficial for cost management in large-scale deployments.
7. You can even use tags(At agent permission level, refer here) to control which roles can be assumed by pods using identity agent.

Points to note about EKS Pod Identity:

1. It requires Amazon EKS cluster version 1.24 or higher, so older clusters may need to be updated before they can leverage this feature. More details here.
2. Applications need to use the latest AWS SDK versions to be compatible with EKS Pod Identity, potentially requiring updates to existing applications.

Conclusion:

Without the need to create/manage OIDC, you can simply provide the permissions to your pods with the EKS identity feature in this simplified way. If you think, this feature is useful for your team(s), it’s the right time to think about it and use it accordingly.

If you liked this article, please show your appreciation by clapping 👏 below! Happy Learning!

One point to note is, AWS charges additionally for providing this capability(object monitoring and automation)

S3 Intelligent-Tiering is the ideal storage class for data with unknown, changing, or unpredictable access patterns, independent of object size or retention period. You can use S3 Intelligent-Tiering as the default storage class for virtually any workload, especially data lakes, data analytics, new applications, and user-generated content.

To check how it works, refer to this link and to know more about costs associated with it, refer to this link.

When to NOT use AWS S3 Intelligent Tiering

Smaller Object Sizes: If you have small object sizes(More percentage of objects smaller than 128KB), then S3 auto-tiering will not be acting upon your objects, as it has a minimum eligible object size of 128 KB. These smaller objects will not be monitored and will always be charged at the Frequent Access tier rates, with no monitoring and automation charges. In this case, you can choose S3 lifecycle management to transfer the objects to lower-cost storage class.

Consistent Access Patterns: If your access patterns are consistent over time and don’t vary significantly, the automatic tiering provided by S3 Intelligent-Tiering might not provide substantial benefits. In such cases, a storage class with a fixed cost structure, like S3 Standard or S3 One Zone-IA, OR configuring a LifeCycle rule will be a better choice.

Data with a Short Lifecycle: If your data has a short lifecycle or if you expect to delete it soon after creation, using S3 intelligent tiering is not the right choice.

Specific Access Requirements: If you have specific access requirements, such as very low-latency access or retrieval within a specific timeframe, you may prefer to keep your data in another high-performance storage class rather than the S3 Intelligent tier.

Simplicity in Cost Management: If you prefer a simpler cost calculation and want to avoid the complexity associated with the automatic tiering process, you can use a single storage class like. Automatic tiering adds complexity to calculations as each automatic storage tier has different pricing plans.

Conclusion

S3 intelligent tiering is a great feature, but at the same time, careful decisions about using it or not, based on the usecase can save more money.

If you liked this article, please show your appreciation by clapping 👏 below! Happy Learning!

During ReInvent-2023, AWS released a new API for AWS Secrets Manager to retrieve the value for a group of secrets in a single API call.

Benefits

1. Offers simplicity for the use cases, where you need to bring multiple secrets into your application.
2. No need to make iterative calls to retrieve one secret at a time.
3. With the BatchGetSecretValue, you can input a list of secret names or ARNs, or filter criteria, such as tags and it will return a response for all matching secrets in the same format as the existing GetSecretValue API.
4. You can reduce the overall amount of API calls and optimize your application.

Demo

Prerequisites:

1. You should have AWS CLI installed in your machine.
2. Should have below mentioned IAM permissions to perform this demo.
3. Should have more than secrets created to retrieve their values.

IAM Permissions:

secretsmanager:BatchGetSecretValue
secretsmanager:GetSecretValue 
secretsmanager:ListSecrets 
kms:Decrypt

Retrieve Multiple Secrets By SecretName:

aws secretsmanager batch-get-secret-value \
    --region RegionName \
    --secret-id-list SecretName1 SecretName2

Output:

{
    "SecretValues": [
        {
            "ARN": "arn:aws:secretsmanager:region:1234567890:secret:SecretName1-abc",
            "Name": "SecretName1",
            "VersionId": "df12ddd-3232-78bv-fdfg",
            "SecretString": "{\"username\":\"admin\",\"password\":\"pass123\"}",
            "VersionStages": [
                "AWSCURRENT"
            ],
            "CreatedDate": "134245351.729"
        },
        {
            "ARN": "arn:aws:secretsmanager:region:1234567890:secret:SecretName2-def",
            "Name": "SecretName2",
            "VersionId": "hj124lkj-3256-56hg-ethf",
            "SecretString": "{\"username\":\"dbuser\",\"password\":\"dbpass123\"}",
            "VersionStages": [
                "AWSCURRENT"
            ],
            "CreatedDate": "143454511.142"
        }
    ],
    "Errors": []
}

Retrieve Multiple Secrets With Filters:

aws secretsmanager batch-get-secret-value \
    --region RegionName \
    --filters Key="name",Values="dbSecret"

It will return all the secrets which has “dbsecret” in the Secret names.

Summary:

A useful and easy-to-integrate feature with minimum changes into your existing code that can save a lot of API calls and can save money if you are making thousands of API calls daily to get different secrets.

If you liked this article, please show your appreciation by clapping 👏 below! Happy Learning!

UseCase:

In this article, we are going to implement the steps to encrypt and decrypt large data(More than 4096 Bytes (4KB)) using AWS KMS Key, which intern uses envelop encryption(Read My Article here).

For encrypting data less than 4KB (Passwords, small files) follow this article.

Prerequisites:

1. AWS Cli should be installed in your personal OR virtual machine.
2. base64 utility should be installed.
3. IAM role OR user credentials should be configured using AWS CLI.
4. AWS KMS Symmetric key should be created using console OR cli OR your own preferred way.
5. Either of ‘KMS Key Policy’ OR ‘IAM role/user policy’ should allow kms GenerateDataKey, encrypt & decrypt permissions (Shown further in the article)
6. Data file with size greater than 4KB.

My Machine Setup:

I have already installed AWS CLI and base64 utility in my EC2 linux machine and my ec2 is already attached with IAM role (Permissions shown below), so I do not need to explicitly follow the step 3 mentioned in prerequisites. Check below:

I have also created the KMS Symmetric key using AWS Console.

As I mentioned above, either my IAM role (attached to EC2) can have below permissions. It is known as identity based policy as it is attached to my identity (IAM Role in this case).

"kms:GenerateDataKey",
"kms:Encrypt",
"kms:Decrypt"

OR, instead of attaching above policy to IAM role/user, you can just attach below policy to the KMS key itself (Known as resource based policy):

{
            "Sid": "Allow use of the key",
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam::1234567890:role/IAMroleName"
            },
            "Action": [
                "kms:Encrypt",
                "kms:Decrypt",
                "kms:ReEncrypt*",
                "kms:GenerateDataKey*",
                "kms:DescribeKey"
            ],
            "Resource": "*"
        }

Doing all this is enough for you to proceed with actual implementation steps.

Implementation (Encryption):

Once all the prerequisites are met, follow below steps for encryption of large data:

#1. Generate Data Key Using KMS

Let’s generate Data key using CMK we generated earlier. It returns Data Key (Plaintext) and Encrypted Data key (CiphertextBlob).

Read more about what is data key here…

aws kms generate-data-key --key-id <kms_key_id> --key-spec AES_256 --region <region>

#2. Decode Base64 encoded Data Key

The Data Key (Plaintext) and Encrypted Data key(CiphertextBlob) generated above are Base64 encoded. Let’s decode just the Plaintext:

echo '<Plaintext_Output_Above>' | base64 --decode > ./plain_data_key.txt

#3. Encrypt Data/file using Plaintext Data Key

Let’s encrypt actual data using Decoded plaintext data key. I have a file with size 8.6KB available in my EC2 instance.

openssl enc -e -aes256 -in data.txt -k plain_data_key.txt > encrypted_data.txt

I get the encrypted_data.txt file as output with encrypted binary data.

#4. Wrapping up steps for encryption

1. Now, the encryption is complete. You can remove your “Plaintext” data key so that it is not misused by anyone.
2. Store your Encrypted data key (CiphertextBlob) (generated as part of step #1) at some safe place (Secret manager etc) so that it can be used in future for decryption.

Implementation (Decryption):

Let’s follow below steps for decryption of encrypted data:

#1. Decode “CiphertextBlob”

echo '<CiphertextBlob>' | base64 --decode > ./encrypted_data_key.txt

#2. Decrypt Encrypted Data Key

Now, we have to decrypt the data key using the Parent KMS key, using which the data key was generated (KMS Symmetric Key in my case).

aws kms decrypt --key-id <kms_key_id> --ciphertext-blob fileb:///root/kms/encrypted_data_key.txt --region <region>

#3. Decode Base64 encoded Plaintext Data Key

Same way as we did for encryption, we have to decode the “Plaintext” key:

echo '<Plaintext_Output_Above>' | base64 --decode > ./decrypted_plain_data_key.txt

#4. Decrypt actual data

Now is the time to decrypt actual data using “decrypted_plain_data_key.txt”

openssl enc -d -aes256 -in encrypted_data.txt -k decrypted_plain_data_key.txt

And I get the data back:

That is it! Now you can again remove the “plaintext” data key.

If you liked this article, please show your appreciation by clapping 👏 below! Happy Learning!

UseCase:

1. You are running your application in ECS with EC2 OR Fargate.
2. You want to run more than one container as part of single task definition and share some particular data (Folders etc) between all the containers.

Example:

1. Let’s say, I have a primary container running my WordPress application and it is generating the log files in the primary container at /myapp/logs location.
2. My second container (Sidecar), has the logs processor running (Example: AWS Cloudwatch, Filebeat etc)
3. My second container wants to access log files generated by my primary container at /myapp/logs location and send them to Cloudwatch/Splunk etc.

Pre-Requisites:

1. You have container-1 and container-2(Sidecar) images ready.
2. You have understanding about ECS task definition to some extent.

Implementation:

Method1 — When you have just one volume to share b/w containers

#1. Modify the volumes section of your task definition

In task definition volumes section, define a data volume with name and DockerVolumeConfiguration values.

"volumes": [{         
"name": "logs",         
"dockerVolumeConfiguration" : 
  {             
  "scope": "task",      #scope can be task or shared      
  "driver": "local"        
  }     
}]

The scope property value can be task or shared. With task scope, the volume is deleted after the task stops. With shared scope, volume persists even if the task stops and persists till the life cycle of the container instance (Example: EC2)

#2. Next step is to make changes in the container definition of container-1 and container-2 (sidecar):

In the containerDefinitions section of container-1 & 2, definemountPoints property that reference the name of the defined volume in the last step and the containerPath to specify the path you want to share.

"containerDefinitions": [{
 "name": "container-1",
 "mountPoints": [{
  "sourceVolume": "logs",
  "containerPath": "/myapp/logs"
 }]
}, {
 "name": "container-2",
 "mountPoints": [{
  "sourceVolume": "logs",
  "containerPath": "/myapp/logs"
 }]
}]

Doing this will mount the same volume to both the containers. If you are running your tasks within ECS with ec2, you can then check the docker volumes created by running docker volume ls command:

That’s it, you can now share the data across containers.

Method2 — When you have more volumes to share b/w containers

Assume, you have more volumes, some of them are local docker volumes, some of them are EBS and EFS volumes used by your primary container. Now if you want to share all of them with your sidecar container aswell, then you can use this method.

#1. It is exactly the same as method one. Only difference is you can define more than one volumes. Example below:

"volumes": [{         
"name": "logs",         
"dockerVolumeConfiguration" : 
  {             
  "scope": "task",      #scope can be task or shared      
  "driver": "local"        
  }     
},
{         
"name": "database",         
"dockerVolumeConfiguration" : 
  {             
  "scope": "shared",      #scope can be task or shared      
  "driver": "local"        
  }     
}
]

#2. Make changes in the container definition of container-1 and container-2 (sidecar):

In the containerDefinitions section of container-1, definemountPoints property that reference the name of the defined volume in the last step and the containerPath to specify the path you want to share.

In the containerDefinitions section of container-2, define volumesFrom property that reference the name of the sourceContainer to share volumes from.

"containerDefinitions": [{
 "name": "container-1",
 "mountPoints": [{
  "sourceVolume": "logs",
  "containerPath": "/myapp/logs"
 },
 {
  "sourceVolume": "database",
  "containerPath": "/var/data"
 }
]
}, 
{
 "name": "container-2",
 "volumesFrom": [{
   "sourceContainer": "container-1",
   "readOnly": false
  }]
}]

Doing this will expose all the volumes used by container-1 with container-2.

If you liked this article, please show your appreciation by clapping 👏 below! Happy Learning!

Background:

In the past, application teams used to store security keys(used for data encryption/decryption) in application configuration files which is a security risk if your key is exposed to intruders, they can decrypt the data.

Introduction:

Envelope encryption is the practice of encrypting plaintext data with a data key (Explained Below), and then encrypting the data key by using another key. In short, the actual data key (Used to encrypt the date) itself is encrypted by some other key so that it can be secured.

In below Diagram:

1. We have a data key (Explained Below), which we use to encrypt data (Data in EBS, S3 or local disk etc)
2. Then we definitely have to secure the data key using which we encrypted the plain format data. That’s why we encrypt the key with another key (In Diagram, AWS KMS key encrypts the data key), and doing this knows as envelop encryption.

Data keys:

1. First of all, data key is just like any other normal key out there (Example: .pem key). Only difference is the algorithm it is generated and the purpose it serves is different.
2. Data keys are symmetric keys you can use to encrypt data, including large amounts of data (Even more than 4KB) and other data encryption keys. For example: EBS, S3, RDS(encryption at rest) is achieved with the help of data keys(Under the hood).
3. Unlike KMS Customer Managed Keys(Not allowed to leave AWS KMS Service), data keys are returned to the user for use outside of AWS KMS.
4. When AWS KMS generates data keys, it returns a plaintext data key for immediate use (To Encrypt the data) and an encrypted copy of the data key that you can safely store with the data. Refer to the image below(Snippet 1.2).
5. When you are ready to decrypt the data, you first ask AWS KMS to decrypt the encrypted copy of the data key (Mentioned in above step) and then actually decrypt the data.
6. AWS KMS has responsibility to only generates, encrypts, and decrypts data keys. Once data key is generated, AWS KMS does not store, manage, or track the data keys. That’s why user have to use and manage data keys outside of AWS KMS.
7. With the help of same AWS KMS key, you can generate as many number of data keys as you want.

In the above snippet (1.2), we have requested to AWS KMS to generate the data key for us (Using KMS Symmetric Key). As a response, it returns:

#1 — Plaintext data key

#2 — Encrypted data key (CiphertextBlob)

How Envelope Encryption works ?

Encryption Process:

1. API request is sent to KMS to generate Data key using CMK. (You can do that using AWS CLI OR SDK).
2. KMS returns response with Plain Data key and Encrypted format Data key.
3. The actual data (EBS, Local disk data) then can be encrypted using Plain Data key.
4. As the data has been encrypted, you can remove the Plain Data key from the memory.
5. Store Encrypted Data Key somewhere safely. It will be used in future to decrypt the data.

Decryption Process:

1. Get the encrypted key from your safe.
2. Send an API request to AWS KMS, to decrypt the key.
3. KMS will return response with Plain Data Key.
4. Decrypt the Encrypted Data using Plain Data key.
5. As the data has been decrypted, you can remove the Plain Data key from the memory.

Summary:

Basically, we have encrypted the key(which we used to encrypt the plain format data) using another key(AWS KMS key in above example). That’s how we achieve envelop encryption. You can do envelop encryption at any number of levels. (Example: First key is encrypted by 2nd and 2nd key is encrypted by 3rd and so on….)

I will perform the demo as part of different article and update the link here.

If you liked this article, please show your appreciation by clapping 👏 below! Happy Learning!

What is KMS Service:

AWS Key Management Service (AWS KMS) is a managed service that makes it easy for you to create and control the cryptographic keys that are used to protect your data. AWS KMS uses hardware security modules (HSM) to protect and validate your AWS KMS keys under the FIPS 140–2 Cryptographic Module Validation Program . Read More…

UseCase:

In this article, we are going to use AWS KMS key to encrypt and decrypt the data which is equal to OR less than 4096 Bytes (4KB). This data size is the limit imposed by AWS (check here….)

Prerequisites:

1. AWS Cli should be installed in your personal OR virtual machine.
2. base64 utility should be installed.
3. IAM role OR user credentials should be configured using AWS CLI.
4. AWS KMS Symmetric key should be created using console OR cli OR your own preferred way.
5. Either of ‘KMS Key Policy’ OR ‘IAM role/user policy’ should allow kms encrypt & decrypt permissions (Shown further in the article)

My Machine Setup:

I have already installed AWS CLI and base64 utility in my EC2 linux machine and my ec2 is already attached with IAM role (Permissions shown below), so I do not need to explicitly follow the step 3 mentioned in prerequisites. Check below:

I have also created the KMS Symmetric key using AWS Console.

As I mentioned above, either my IAM role (attached to EC2) can have below permissions. It is known as identity based policy as it is attached to my identity (IAM Role in this case).

"kms:Encrypt",
"kms:Decrypt"

OR, instead of attaching above policy to IAM role/user, you can just attach below policy to the KMS key itself (Known as resource based policy):

{
            "Sid": "Allow use of the key",
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam::1234567890:role/roleName"
            },
            "Action": [
                "kms:Encrypt",
                "kms:Decrypt",
                "kms:ReEncrypt*",
                "kms:GenerateDataKey*",
                "kms:DescribeKey"
            ],
            "Resource": "*"
        }

Doing all this is enough for you to proceed with actual implementation steps.

Implementation:

Once all the prerequisites are met, follow below steps for encryption and decryption:

#1 Create a file with some data:

On your machine (In my case Linux machine), create a text file or a file of your choice. I have named this file as “data.txt” with below data. Make sure that the file size is less than 4KB:

In my case the file size is even less than 1KB.

#2 Encrypt the File:

Run below command to encrypt the file:

#Run Below From Linux Machine:
aws kms encrypt \
    --key-id myKMSKeyId \
    --plaintext fileb://data.txt\
    --region region \
    --output text \
    --query CiphertextBlob | base64 \
    --decode > encrypted-data.txt

##############################################
#Run Below From Windows:

aws kms encrypt \
    --key-id myKMSKeyId \
    --plaintext fileb://data.txt\
    --output text \
    --query CiphertextBlob > C:\Temp\encrypted-data.base64

certutil -decode C:\Temp\encrypted-data.base64 C:\Temp\encrypted-data

Below is the output after I executed above command(In Linux Machne). As you see I get the encrypted-data.txt file generated and having encrypted data within it!

--plaintext: Uses the --plaintext parameter to indicate the data to encrypt. This parameter value must be base64-encoded. or you must use the fileb:// in file name, so that data is automatically converted into binary data.

--output and --query: Ideally, running the kms encrypt command returns many things(a json object). These two parameters(output and query) are used to just extract the encrypted data and nothing else.

base64: Uses the base64 utility to decode the extracted output into binary data.

> encrypted-data.txt: The final part of the command ( > encrypted-data.txt) saves the binary ciphertext to a file to make decryption easier.

#3 Decrypt the File:

#Run Below From Linux Machine:

aws kms decrypt \
    --ciphertext-blob fileb://encrypted-data.txt \
    --key-id myKMSKeyId \
    --region region \
    --output text \
    --query Plaintext | base64 \
    --decode > myplaintext.txt

##############################################

#Run Below From Windows Command Prompt:

aws kms decrypt ^
    --ciphertext-blob fileb://encrypted-data.txt^
    --key-id myKMSKeyId ^
    --region region ^
    --output text ^
    --query Plaintext > myplaintext.base64

certutil -decode myplaintext.base64 myplaintext

Below is the output after I executed above command (In Linux Machine). As you see I get the myplaintext.txt file generated and having decrypted data within it!

-ciphertext-blob parameter, use the fileb:// prefix, which tells the CLI to read the data from a binary file.

Rest all the properties are same as encryption part.

Can I perform all the above with KMS — Asymmetric key aswell?

Yes, you can do the same with AWS KMS — Asymmetric key also. You just need to provide one extra command line parameter that is “ — encryption-algorithm” while running the command.

What if my file size is more than 4KB?

Lets see….

I will create a file of size 10kb.

If I now run the encrypt command then?

I get the validation exception, thrown by AWS.

So does this mean that KMS cannot handle data more than 4KB? No, its not the case, there is way known by name “Envelope Encryption”(Read My Article Here).

If you want to read about what is Envelope Encryption you can read that here.

If you liked this article, please show your appreciation by clapping 👏 below! Happy Learning!

1. Assume you are developing an application in Java (OR .Net, PHP etc) OR you have a vendor developed application.
2. As you know, for our application/middlewares to work, we need some configuration files (For example: Nginx/apache/jboss config file OR some other application specific configuration files) containing application configuration parameters, DB connection details etc.
3. You must be storing your application code in some git repository of your choice. However, you would not like to keep your configuration details hardcoded in the git repo (Because it is either some secret values OR those values are changing with each environment like dev/test/prod)
4. Your application is not written in a way to dynamically fetch those values from key-value store (Example: AWS Secrets manager, DynamoDB etc)
5. To have the configuration files dynamically populated with right set of values, this article can suggest you few solutions.

Solution Explanation:

In below diagram, you can see we have three stages, inputs, Template Rendering and Output.

#1 Inputs stage: In this stage we need input configuration file(s) having placeholders (See below example config file with yellow highlighted placeholders) and the actual values. These actual values can be provided from any source (Example: OS level Env variables, AWS secrets Manager, DynamoDB, consul server etc)

#2 TemplateRendering Stage: In this stage, the input (Config file with placeholders and the actual values) are processed by the use of tools like confd, consul-template or Ansible or any other tool available in the market.

#3 Output Stage: Here we get the final rendered configuration files placed at required location within virtual machine OR docker container. Which can be used the application to work properly.

Real UseCase Example:

Earlier in one of our application(s), we were building the Docker images and running them in AWS ECS orchestrator. we used confd to render the application config files. We wanted to avoid having the configuration files with real values in the GIT as well as in the Docker Image.

So we kept the Dockerfile and other application configuration files(Just templates with placeholders) in the Git repository and built the docker image using the same and provided the real values to the container at runtime(as OS environment variables) using AWS Secrets Manager (through AWS ECS & Secrets Manager default Integration provide by AWS) and confd(At runtime) rendered the templates and kept them at required location within container and we started the application using Docker entrypoint. (Refer Below Diagram)

If I put it in terms of three stages then below is how it looks:

1. Input Stage: Configuration files stored in Git Repo and actual values stored in the AWS Secrets manager.
2. Render Stage: Confd used for rendering within Docker container.
3. Output Stage: Config files with real values generated and placed within container at runtime.

Note: If we use consul-template for above use case instead of confd, it will work exactly the same way, without any issues.

Comparison Confd/Consul-Template/Ansible:

Confd:

confd is a lightweight configuration management tool focused on:

• keeping local configuration files up-to-date using data stored in etcd, consul, dynamodb, redis, vault, zookeeper, aws ssm parameter store or env vars and processing template resources.
• reloading applications to pick up new config file changes

More Information…………

Confd Pros:

1. It is very lightweight and Open Source tool, just a simple binary is need to do the job.
2. We can make our application environment independent (dev/test/prod etc)
3. It supports jinja like conditioning (if/else, loops) as part of writing templates.
4. We do not need to expose secrets and other important configuration values in Git repo or the docker image or the virtual machine.
5. confd supports many data sources (etcd, redis, AWS SSM, Vault etc).
6. It can look for dynamic changes in the application configuration.

Confd Cons:

1. The confd new versions are not released from last 5 years. So it is not actively maintained.

Consul-Templates:

consul-template provides convenient way to populate values from Consul into the file system using the consul-template daemon.

The daemon consul-template queries a Consul, Vault, Nomad cluster, or OS environment variables and updates any number of specified templates on the file system. As an added bonus, it can optionally run arbitrary commands when the update process completes.

More Information…………

Consul-Template Pros:

1. It is very lightweight and Open Source tool developed and maintained by HashiCorp. New versions are being released every couple of months.
2. We can make our application environment independent (dev/test/prod etc)
3. We do not need to expose secrets and other important configuration values in Git repo or the docker image or the virtual machine.
4. It is feature rich tool mainly if you are using HashiCorp’s other tooling.
5. It can look for dynamic changes in the application configuration.

Consul-Template Cons:

1. It supports fewer data sources (Consul, Vault, or Nomad). It is mainly built to have integration with Consul and other HashiCorp solution.

Ansible:

Ansible is a radically simple IT automation system. It handles configuration management, application deployment, cloud provisioning, ad-hoc task execution, network automation, and multi-node orchestration. Ansible makes complex changes like zero-downtime rolling updates with load balancers easy.

More Information…………

Ansible-Template Pros:

1. It is Open Source, Simple, easy to learn and well maintained, python based configuration management tool.
2. It is feature rich tool and has integration with plenty of tools and can be used to serve plenty of use cases.
3. We can make our application environment independent (dev/test/prod etc)
4. We do not need to expose secrets and other important configuration values in Git repo or the docker image or the virtual machine.

Ansible Cons:

1. If we talk about just configuration rendering (Like this article is intended for) then Ansible is not the right choice. You will have to do more to achieve the job. Ansible is a perfect solution for other complex use cases like if you want to configure multiple machines with some set of softwares, if you want to configure and deploy your application OR orchestrate the Cloud infra to name a few. It is advances configuration management tool not meant for just rendering templates.

From HashiCorp perspective, consul-template is different from confd and ansible in following way:

Source: this

Conclusion:

1. All the three tools are efficient to do the job (Configuration rendering), your selection totally depends on your use case.
2. As such, Ansible is meant to do a lot and its pretty advanced tool to serve different use cases. But if you are just looking for your configuration rendering and a light weight solution then confd and consul-template are the right choice.

If you liked this article, please show your appreciation by clapping 👏 below! Happy Learning!

Please Note: This article assumes that you have good understanding about EC2 services and the traditional ways to open session with EC2 Linux and Windows instances.

UseCase Explanation:

1. Ideally, instance in private subnets are not directly accessible if you are not using VPN/DirectConnect connectivity.
2. In past, we used to have bastion host setup in public subnet and ssh tunneling to connect to private subnet instances.
3. In this article, we will see the more secure way to connect your instances in AWS private subnets using AWS session manager.

Prerequisites:

For this article, I have used Windows laptop and Powershell as the scripting language to initiate the remote sessions. You can also use Windows bash or GitBash cli tools.

You must have the following tools and services installed on your personal computer:

• The AWS Command Line Interface (AWS CLI).
• The Session Manager plugin installed locally
• IAM user with programmatic access. (Mainly session manager permissions).

You also need the following AWS components and services:

• Linux & Windows EC2 instance in the private subnet.
• SSM agent installed in the EC2 instance. (It also comes pre-installed by default in few AWS provided AMI’s like AWS Linux2. Refer this). Follow these instructions to install the SSM agent.
• Required IAM permissions added to the new/existing IAM role. Follow this for permissions.
• You EC2 SG should have outbound connectivity on port 443 to Systems Manager endpoints. So that SSM agent can communicate with the sessions manager API. OR you can also keep the outbound default rule which allows all traffic.

Implementation:

#1 Once you have EC2(Linux & Windows) in private subnet with all the prerequisites (SSM agent, IAM role and SG setup), you should see the EC2 instance shown the AWS Systems Manager ===> Session manager service

#2 (Optional) You can connect to your private EC2 instance from above screen, and it will open browser based CLI session for you.

#3 To connect to private EC2 instance (Linux) from your local computer, run the following command

aws ssm start-session --target <instance-id> --region <region>

It will open the session and thats it for Linux instance!

#4 To connect to private EC2 instance (Windows) from your local computer, run the following command:

aws ssm start-session --target <instance-id> --document-name AWS-StartPortForwardingSession --parameters "localPortNumber=54321,portNumber=3389" --region <region>

It will open the session and show below screen:

Leave this window as it is!

Open RDP session window (You can grab the administrator user password from the AWS console with your EC2 private keypair)

Enter the Administrator password and doing this will open the GUI based session for your Windows server and that is it.

#5 What If you lost the Windows private key pair?

If you loose your Windows private key, then you can open CLI based session with windows instance, exactly the same way you do with Linux:

aws ssm start-session --target <windows-instance-id> --region <region>

It will open the CLI session for you and you can add Windows user and add the user to the Administrator group(Refer This) and then follow the same process of AWS session manager StartPortForwardingSession and again open GUI based RDP session.

If you liked this article, please show your appreciation by clapping 👏 below! Happy Learning!