Open in app

Sign in

Write

Sign in

Managing MongoDB Atlas & AWS PrivateLink with Terraform Modules

Prashant Vyas
10 min readOct 4, 2023

--

In the ever-evolving world of DevOps and cloud-native architecture, the ability to provision and manage infrastructure is the key. That’s where Infrastructure as a Code and Terraform come into play. In this post, we will discuss the provisioning of MongoDB, one of the most popular NoSQL databases, using Terraform modules.

We use Terraform to deploy and configure MongoDB Atlas provides a structured, repeatable, and version-controlled approach to managing your database infrastructure.

A Terraform module is a reusable set of instructions for creating and managing infrastructure components. In this blog, we’ll explore how to use Terraform Modules to automate MongoDB Atlas provisioning.

Considering we have our applications and services running out of AWS cloud which would be connecting to Atlas MongoDB cluster. So it is assumed a relevant AWS VPC with subnets already exists. We will be creating Mongo Cluster, AWS PrivateLink, Security groups etc. For the deployment we are using an M10 Cluster Tier (which is 10GB storage, 2GB memory and 2vCPU on AWS) you may configure based on your needs.

Resources

MongoDB Atlas — Resource: MongoDB Atlas Cluster
MongoDB Atlas Cluster is a fully managed database cluster hosted on MongoDB’s cloud infrastructure. It provides a scalable, reliable, and secure MongoDB database environment. MongoDB with its document-oriented data model and scalability features has become an essential part of modern applications. Some of the features are

  • Auto-Scaling: Scales horizontally and vertically based on workload.
  • Automated Backups: Regular automated backups and point-in-time recovery.
  • Security: Built-in security features, including network isolation, encryption, and role-based access control.
  • Monitoring: Real-time performance monitoring and alerting.
  • Global Clusters: Deploy clusters in multiple regions for high availability and low-latency access.

MongoDB Atlas Project — Resource: MongoDB Atlas Project
MongoDB Atlas Project is a logical container for MongoDB Atlas resources. It helps organize and manage your clusters, teams, and access controls. Some of the features are

  • Resource Organization: Group clusters and other resources into projects for easier management.
  • Team Collaboration: Collaborate with team members and assign roles and permissions.
  • Billing and Cost Control: Manage billing and set spending limits for projects.
  • Network Peering: Connect clusters within a project or across projects securely

AWS VPC— Resource: VPC (Virtual Private Cloud)
AWS Virtual Private Cloud (VPC) is a logically isolated section of the AWS cloud where you can launch AWS resources in a defined network space. Some of the features are

  • Isolation: Provides network isolation for your AWS resources.
  • Customization: Customize IP address ranges, route tables, and network gateways.
  • Security: Control access to resources within your VPC using security groups and network ACLs.
  • Connectivity Options: Establish connectivity between VPCs using VPC peering, VPN, or AWS Direct Connect.

AWS PrivateLink — Resource: AWS PrivateLink
AWS PrivateLink is a network service that enables secure and private communication between your VPC and supported AWS services or endpoints without traversing the public internet.

  • Private Connectivity: Access AWS services privately over the AWS global network.
  • Enhanced Security: Keep traffic within your VPC, reducing exposure to public internet threats.
  • Simple Setup: Configure PrivateLink endpoints to enable private connectivity.
  • Integrated Services: Supports a growing list of AWS services, including Amazon S3, DynamoDB, and more

Prerequisites

  1. Basic Knowledge: Ensure you have a fundamental understanding of MongoDB Atlas and Terraform. Familiarity with MongoDB Atlas concepts like clusters, projects, and organization setup will be beneficial.
  2. MongoDB Atlas Account: You’ll need an active MongoDB Atlas account. If you haven’t set up one yet, visit MongoDB Atlas to create an account, set up an organization, create a Project under the Organization and configure any necessary security settings. To set up a private endpoint, you must have Organization Owner or Project Owner access to the project.
  3. Generate MongoDB API Keys (Public and Private) for the user under the project which we will be using for creating resources
  4. Terraform should be installed on your local machine. If you haven’t already installed it, follow the installation instructions for your platform on the official Terraform website.

Creating Terraform Configuration

Now that we have our prerequisites in place, it’s time to create the Terraform configuration to provision our MongoDB Atlas resources. We will be using Terraform modules to keep the configuration organized. We will define and provision MongoDB Atlas Clusters and set up AWS PrivateLink connectivity using Terraform.

TF Structure



├── env
│   └── dev
│       └── terraform.tfvars
├── modules
│   ├── mongodb_atlas_cluster
│   │   ├── atlas-cluster.tf
│   │   ├── variables.tf
│   │   └── versions.tf
│   └── mongodb_privatelink
│       ├── mongodb-pvtlink.tf
│       ├── securitygroup.tf
│       ├── variables.tf
│       └── versions.tf
├── mongodb-atlas-cluster.tf
├── mongodb-pvtlink.tf
├── output.tf
├── provider.tf
└── variables.tf
  • env/` Directory: This directory contains environment-specific configurations, such as dev/ for the development environment. It typically includes a terraform.tfvars file to set environment-specific variable values
  • modules/ Directory: The modules/ directory houses custom Terraform modules. In our project, we have two modules: mongodb_atlas_cluster and mongodb_privatelink. Each module has its own configuration files, including variables.tf, versions.tf, and module-specific Terraform files (atlas-cluster.tf and mongodb-pvtlink.tf).
  • Top-level Configuration Files: The top-level of the project directory contains configuration files that coordinate the use of modules and define the infrastructure layout. These files include mongodb-atlas-cluster.tf, mongodb-pvtlink.tf, provider-mongodb-cluster.tf, variables.tf, and output.tf

Storing API keys
Security is crucial to protect your MongoDB resources. You can store your API keys in safe and manageable ways using environment variables and a tool like HashiCorp Vault. Here’s how you can do it:
1. Environment Variables: Store your MongoDB Atlas API keys as environment variables on your local machine.

export MONGODB_ATLAS_PUBLIC_KEY=your_public_key
export MONGODB_ATLAS_PRIVATE_KEY=your_private_key

To load these environment variables into your Terraform configurations you can reference them in your provider as mentioned below in provider.tf

Let’s explore the contents of these configuration files in more detail.

Top-level Configuration Files

1. Provider
In this file, we configure the MongoDB Atlas provider. It specifies how to authenticate with MongoDB Atlas using the API keys stored in environment variables or HashiCorp Vault — provider.tf

terraform {
  required_version = ">= 0.14.11"
  required_providers {
    mongodbatlas = {
      source  = "mongodb/mongodbatlas"
      version = "1.0.1"
    }
  }
}

provider "mongodbatlas" {
  alias       = "cluster"
  public_key  = var.mongodb_atlas_public_key
  private_key = var.mongodb_atlas_private_key
}

2. MongoDB Cluster
The mongodb_atlas_cluster.tf file instantiates a Terraform module called mongodb_atlas_cluster. This module is responsible for provisioning MongoDB Atlas clusters. It takes various configuration variables such as cluster name, project name, version, and tier from the terraform.tfvars file. The module uses these variables to create MongoDB Atlas clusters with the specified settings, utilizing the MongoDB Atlas API keys for authentication.

module "mongodbatlas_cluster" {
  source                   = "./modules/mongodb_atlas_cluster"
  cluster_name             = var.mongodbatlas_cluster.cluster_name
  project_name             = var.mongodbatlas_cluster.project_name
  mongo_db_major_version   = var.mongodbatlas_cluster.mongo_db_major_version
  cluster_tier             = var.mongodbatlas_cluster.cluster_tier
  mongodb_atlas_api_key    = var.mongodb_atlas_private_key
  mongodb_atlas_public_key = var.mongodb_atlas_public_key
  replication_factor       = var.mongodbatlas_cluster.replication_factor
  electable_nodes          = var.mongodbatlas_cluster.replication_specs.regions_config.electable_nodes
  providers = {
    mongodbatlas = mongodbatlas.cluster
  }
}

4. PrivateLink
The mongodb-pvtlink.tf file instantiates a Terraform module called mongodb-pvtlink. This module is responsible for configuring AWS PrivateLink connectivity for MongoDB Atlas. It takes various configuration variables such as PrivateLink name, tags, project ID, VPC ID, subnet IDs, egress, and ingress rules from the terraform.tfvars file. The module uses these variables to set up secure private connectivity between your VPC and MongoDB Atlas, enhancing network security and isolation. This approach simplifies the configuration of AWS PrivateLink for MongoDB Atlas within your Terraform code.

module "mongodb-pvtlink" {
  source               = "./modules/mongodb_privatelink"
  mongodb_pvtlink_name = var.mongodb_awspvt_endpoints.mongodb_pvtlink_name
  tags                 = var.tags
  atlasprojectid       = var.mongodb_awspvt_endpoints.atlasprojectid
  primary_vpc_id       = var.primary_vpc_id
  subnet_ids           = var.subnet_ids
  egress               = var.mongodb_awspvt_endpoints.egress
  ingress              = var.mongodb_awspvt_endpoints.ingress
  providers = {
    mongodbatlas = mongodbatlas.cluster
  }
}

5. Module-1
modules/mongodb_atlas_cluster
The atlas-cluster.tf file defines a Terraform resource block for creating a MongoDB Atlas cluster. It uses variables like cluster_name, mongo_db_major_version, and replication_factor to configure the cluster's properties. The resource specifies settings such as cloud backup, auto-scaling, and replication specifications. Additionally, it retrieves the project ID using data sources from MongoDB Atlas based on the provided project name.

resource "mongodbatlas_cluster" "cluster" {
  name                         = var.cluster_name
  project_id                   = data.mongodbatlas_project.project.id
  mongo_db_major_version       = var.mongo_db_major_version
  provider_name                = "AWS"
  provider_region_name         = "US_EAST_1"
  provider_instance_size_name  = var.cluster_tier
  cloud_backup                 = true
  auto_scaling_disk_gb_enabled = true
  replication_factor           = var.replication_factor
  cluster_type                 = "REPLICASET"

  replication_specs {
    num_shards = 1
    regions_config {
      region_name     = "US_EAST_1"
      priority        = 7
      electable_nodes = var.electable_nodes
    }
  }
}

data "mongodbatlas_project" "project" {
  name = var.project_name
}

Module-2
modules/mongodb_privatelink/mongodbpvtlink.tf

####### MongoDB Private Link for Connectivity with Application
resource "mongodbatlas_privatelink_endpoint" "atlas-pl-ep" {
  project_id    = var.atlasprojectid
  provider_name = "AWS"
  region        = var.aws_region
}

resource "mongodbatlas_privatelink_endpoint_service" "atlas-ep-svc" {
  project_id          = mongodbatlas_privatelink_endpoint.atlas-pl-ep.project_id
  endpoint_service_id = aws_vpc_endpoint.mongodb-endpoint.id
  private_link_id     = mongodbatlas_privatelink_endpoint.atlas-pl-ep.private_link_id
  provider_name       = "AWS"
}

# AWS VPC endpoint
resource "aws_vpc_endpoint" "mongodb-endpoint" {
  vpc_id             = var.primary_vpc_id
  service_name       = mongodbatlas_privatelink_endpoint.atlas-pl-ep.endpoint_service_name
  vpc_endpoint_type  = "Interface"
  subnet_ids         = var.subnet_ids
  security_group_ids = [aws_security_group.mongodb-pvtlink.id]
  tags               = merge(var.tags, { "Name" : var.mongodb_pvtlink_name })
  auto_accept        = true
}

file configures MongoDB Atlas PrivateLink connectivity with an application. It defines three key resources:

  1. mongodbatlas_privatelink_endpoint: This resource establishes a PrivateLink endpoint for MongoDB Atlas, specifying the project ID and AWS region.
  2. mongodbatlas_privatelink_endpoint_service: This resource connects the PrivateLink endpoint to an AWS VPC endpoint, ensuring secure communication between MongoDB Atlas and the VPC.

aws_vpc_endpoint: This resource creates an AWS VPC endpoint, setting the VPC ID, service name, subnet IDs, security group, and tags. It allows for interface-based connectivity to MongoDB Atlas via PrivateLink

Module-3
modules/mongodb_privatelink/securitygroup.tf

# AWS Security Groups
resource "aws_security_group" "mongodb-pvtlink" {
  name        = var.mongodb_pvtlink_name
  description = "Security group for mongoDB connectivity from AWS platform ${var.primary_vpc_id}"
  vpc_id      = var.primary_vpc_id
  ingress {
    from_port       = var.ingress.from_port
    to_port         = var.ingress.to_port
    protocol        = var.ingress.protocol
    security_groups = var.ingress.security_groups
  }
  egress {
    from_port   = var.egress.from_port
    to_port     = var.egress.to_port
    protocol    = var.egress.protocol
    cidr_blocks = var.egress.cidr_blocks
  }
}

For each resource that needs to connect to your Atlas clusters using AWS PrivateLink, the resource’s security group must allow outbound traffic to the interface endpoint’s private IP addresses on all ports. Security group ensures that the necessary network traffic is allowed for secure communication between your AWS resources and MongoDB Atlas via PrivateLink while maintaining control over the ingress and egress traffic it includes.

  1. aws_security_group "mongodb-pvtlink": This resource creates an AWS security group named based on the provided mongodb_pvtlink_name. It's intended for securing MongoDB connectivity from the AWS platform within the specified VPC (var.primary_vpc_id).
  2. Ingress Rules: It configures ingress rules to allow incoming traffic based on the settings provided in the var.ingress variables, specifying the source security groups and ports.
  3. Egress Rules: It also defines egress rules to control outgoing traffic based on the settings in the var.egress variables, specifying the destination CIDR blocks and ports

7. Variables:
This file defines variables that can be used throughout the project. These variables help customize the behaviour of the modules and configuration files — variables.tf

variable "atlasprojectid" {
  description = "Atlas project ID"
  type        = string
}

variable "aws_region" {
  type    = string
  default = "US_EAST_1"
}

variable "tags" {
  description = "TAGS on AWS"
  type        = map(string)
}

variable "primary_vpc_id" {
  description = "VPC ID"
  type        = string
}

variable "subnet_ids" {
  description = "subnet ids"
  type        = list(string)
}

variable "mongodb_pvtlink_name" {
  description = "Name of the Private Link"
  type        = string
}

variable "egress" {
  description = "mongodb Egress"
  type = object({
    from_port   = number
    to_port     = number
    protocol    = string
    cidr_blocks = list(string)
  })
}

variable "ingress" {
  description = "mongodb Egress"
  type = object({
    from_port       = number
    to_port         = number
    protocol        = string
    security_groups = list(string)
  })
}

Environment-specific Configuration

The env/ directory is where you can customize settings for different environments. For example, env/dev/terraform.tfvars you might set variables specific to the development environment.

region         = "us-east-1"
primary_vpc_id = "vpc-<xxxxxxxxxx>"

tags = {
  "environment"   = "Dev"
  "business_unit" = "Test Atlas Mongo Cluster"
}
#################################  MongoDB ###################
######## Atlas Project API Keys
mongodb_atlas_public_key  = "<xxxxx>"
mongodb_atlas_private_key = "<xxxxxx-xxxx-xxxx-xxxx-xxxxxxx>"

#### ------ MongoDB Clusters
mongodbatlas_cluster = {
  cluster_name           = "<cluster NAME>"
  project_name           = "<project NAME>"
  atlasprojectid         = "<project ID>"
  cluster_tier           = "M10"
  mongo_db_major_version = "4.4"
  provider_region_name   = "US_EAST_1"
  replication_factor     = 3
  replication_specs = {
    num_shards = 1
    regions_config = {
      priority        = 7
      electable_nodes = 3
    }
  }
}

#### ------- MongoDB Private link
mongodb_awspvt_endpoints = {
  mongodb_pvtlink_name = "mongodb-awspvtlink"
  atlasprojectid       = "<project ID>"
  egress = {
    from_port   = 1024
    to_port     = 1073
    protocol    = "TCP"
    cidr_blocks = ["10.0.101.0/24"]  // Private subnet
  }
  ingress = {
    from_port       = 0
    to_port         = 65535
    protocol        = "tcp"
    security_groups = ["sg-<xxxxxxxxxx>"] //Security Group
  }
}

Apply Terraform

terraform fmt --recursive //formatting
terraform init 
terraform plan -var-file=/path/to/terraform.tfvars
terraform apply -var-file=/path/to/terraform.tfvars
  1. terraform fmt --recursive: Formats all Terraform configuration files in the directories, ensuring consistent code style
  2. terraform init: Initializes the Terraform project by downloading the required providers and modules
  3. terraform plan: Generates an execution plan that shows the changes Terraform will make to the infrastructure without actually applying
  4. terraform apply: Applies the Terraform configuration to provision or modify infrastructure according to the defined plan

This will usually take about 10–12 minutes you will see ‘Apply complete! Resources: 4 added, 0 changed, 0 destroyed.’

Verify Resources

Destroy Terraform

terraform destroy

Finally you can destroy all the resources that you have created.

References
Mongo Network Security
Mongo Cluster
Private Endpoints in Atlas
AWS PrivateLink

Let me know if you have any queries. Please provide your feedback and don’t forget the 👏✌️❤️ if you like this content!
You can follow me on Linkedin

Thank You!

Terraform
AWS
Infrastructure As Code
Security
Mongodb

--

--

Prashant Vyas

Written by Prashant Vyas

7 Followers

Help

Status

About

Careers

Press

Blog

Privacy

Terms

Text to speech

Teams