yavarin.tech

I previously wrote a post on how Terraform remote backend can help us to work more collaboratively on the same Terraform code. Make sure you read it here, before continuing this post to find out what is a Terraform remote backend and why you might need to switch to it and get familiar with the terminologies. In this post I am trying to demonstrate how to configure the Terraform remote backend and what are the considerations you might want to keep into account while adding this piece to your infrastructure.

The ingredients 🧑‍🍳

In the example that I give here I am using AWS as my cloud provider so for the backend I’ll be using S3 storage services and for the locking mechanisms I am using AWS DynamoDB tables. You can find the code I am using in this post on my GitHub.

The code

I am deploying an EC2 instance to the AWS cloud. My code would look like this:

terraform {
  required_providers {
    aws = {
      source  = "hashicorp/aws"
    }
  }

  required_version = ">= 1.9.0"
}

provider "aws" {
  profile = "devops-projects"
  region  = "us-east-1"
}

resource "aws_default_vpc" "default" {
  tags = {
    Name = "Default VPC"
  }
}

resource "aws_instance" "ec2-instance-1" {
  ami           = "ami-03972092c42e8c0ca"
  instance_type = "t2.micro"
  tags = {
    Name = "remote-backend-lab"
  }
}

If you ever have executed any terraform code you’ve noticed that after applying the code terraform will add to files to your working directory. Specifically I am talking about the terraform.tfstate and the terraform.tfstate.backup files. These files serve a specific purpose. The terraform.tfstate is used to store your infrastructure latest state and will be read and updated every time you run terraform plan or terraform apply commands. If it does not exist it means your infrastructure does not exist in terraform and if you try to run the terraform plan or terraform apply commands you would see that terraform is not able to detect your configurations. The terraform.tfstate.backup file serves as a backup as your infrastructure is important, you can read more about these files here. I our example to make sure that every developer can have access to the state files we can share it with them in a github repository or a S3 bucket. This way developers who work with the infrastructure code are able to see what is already deployed and they can provision their resources without removing other’s work. But there is a problem with this method. Let’s see what happens if two developers who are working on the same infrastructure code want to create an EC2 instance.

Please note that the terraform state files can contain clear text sensitive data so take security measures while storing and sharing these file. You can use git-crypt or S3 server side encryption for example.

Developer A pulls the main.tf file alongside with the terraform.tfstate and the terraform.tfstate.backup files from the git repository and adds his own changes to the file.

terraform {
  required_providers {
    aws = {
      source  = "hashicorp/aws"
    }
  }

  required_version = ">= 1.9.0"
}

provider "aws" {
  profile = "devops-projects"
  region  = "us-east-1"
}

resource "aws_default_vpc" "default" {
  tags = {
    Name = "Default VPC"
  }
}

resource "aws_instance" "ec2-instance-1" {
  ami           = "ami-03972092c42e8c0ca"
  instance_type = "t2.micro"
  tags = {
    Name = "remote-backend-lab"
  }
}

resource "aws_instance" "developer-a-ec2-instance" {
  ami           = "ami-03972092c42e8c0ca"
  instance_type = "t2.micro"
  tags = {
    Name = "Developer_A_instance"
  }
}

In the mean time Developer be does the same and adds his changes to the main.tf file.

terraform {
  required_providers {
    aws = {
      source  = "hashicorp/aws"
    }
  }

  required_version = ">= 1.9.0"
}

provider "aws" {
  profile = "devops-projects"
  region  = "us-east-1"
}

resource "aws_default_vpc" "default" {
  tags = {
    Name = "Default VPC"
  }
}

resource "aws_instance" "ec2-instance-1" {
  ami           = "ami-03972092c42e8c0ca"
  instance_type = "t2.micro"
  tags = {
    Name = "remote-backend-lab"
  }
}

resource "aws_instance" "developer-b-ec2-instance" {
  ami           = "ami-03972092c42e8c0ca"
  instance_type = "t2.micro"
  tags = {
    Name = "Developer_B_instance"
  }
}

As you can see:

  1. The main.tf file on each developers local file system are different.
  2. They are using a state file that does not include the other’s changes. Given the above observations every time one of the developers deploy their codes to the infrastructure using terraform apply it will delete the instance that is created by the other developer. Here is where the Terraform remote backend comes handy! Let’s see how it will help us solving this issue in action.

First we need to prepare the infrastructure

As mentioned above we are going to use a S3 bucket and a DynamoDB table for this post. But before adding the configurations for the remote backend we need to create th resources. Below is hour initial terraform code after adding the required resources for creating a DynamoDB table plus a S3 bucket.

terraform {
  required_providers {
    aws = {
      source  = "hashicorp/aws"
    }
  }

  required_version = ">= 1.9.0"
}

provider "aws" {
  profile = "devops-projects"
  region  = "us-east-1"
}

resource "aws_default_vpc" "default" {
  tags = {
    Name = "Default VPC"
  }
}

resource "aws_instance" "ec2-instance-1" {
  ami           = "ami-03972092c42e8c0ca"
  instance_type = "t2.micro"
  tags = {
    Name = "remote-backend-lab"
  }
}

resource "aws_kms_key" "bucketkey" {
  description = "This key is used to encrypt the bucket data"
  deletion_window_in_days = 7
}

resource "aws_s3_bucket" "terraform_state" {
  bucket = "devopsprojects-terraformstate"
}

resource "aws_s3_bucket_server_side_encryption_configuration" "terraform_state" {
  bucket = aws_s3_bucket.terraform_state.id
  rule {
    apply_server_side_encryption_by_default {
      kms_master_key_id = aws_kms_key.bucketkey.arn
      sse_algorithm = "aws:kms"
    }
  }
}
resource "aws_s3_bucket_versioning" "terraform_state" {
  bucket = aws_s3_bucket.terraform_state.id
  versioning_configuration {
    status = "Enabled"
  }
}

resource "aws_s3_bucket_lifecycle_configuration" "terraform_state" {
  bucket = aws_s3_bucket.terraform_state.id

  rule {
    id      = "Prevent Accidental Deletion"
    status = "Enabled"

    noncurrent_version_expiration {
      noncurrent_days = 365
    }
    abort_incomplete_multipart_upload {
      days_after_initiation = 7
    }
    }
  }

resource "aws_s3_bucket_public_access_block" "terraform_state" {
  bucket = aws_s3_bucket.terraform_state.id

  block_public_acls       = true
  block_public_policy     = true
  ignore_public_acls      = true
  restrict_public_buckets = true
}

resource "aws_dynamodb_table" "terraform_locks" {
  name         = "terraform-locks"
  billing_mode = "PAY_PER_REQUEST"
  hash_key     = "LockID"

  attribute {
    name = "LockID"
    type = "S"
  }
}

As you can see I enabled versioning on the bucket so I will always have the history of changes of the state file.You can configure the S3 bucket as per your requirements the example given here might not be suitable for production environments. Also you need to make sure the the attribute name is exactly LockID and of type string so terraform can pick it up.

Now we add the remote backend configs

After applying the code to our infrastructure, we are ready to configure terraform to use these services as remote backend to store the sate file. We need to add the config to the terraform block of the main.tf file like below.

terraform {
  required_providers {
    aws = {
      source  = "hashicorp/aws"
    }
  }
  backend "s3" {
    bucket         = "devopsprojects-terraformstate"
    key            = "devops-projects/us-east-1/terraform.tfstate" #A path to store the terraform state file in the bucket
    region         = "us-east-1"
    dynamodb_table = "terraform-locks"
    profile = "devops-projects"

  }
  required_version = ">= 1.9.0"
}
.
.
. #rest of the code

Now if we execute the terraform apply command we will get a prompt asking to initialize the backed. Because we are trying to save the state file now in the AWS S3 bucket that we created previously.

│ Error: Backend initialization required, please run "terraform init"
│
│ Reason: Initial configuration of the requested backend "s3"
│
│ The "backend" is the interface that Terraform uses to store state,
│ perform operations, etc. If this message is showing up, it means that the
│ Terraform configuration you're using is using a custom configuration for
│ the Terraform backend.
│
│ Changes to backend configurations require reinitialization. This allows
│ Terraform to set up the new configuration, copy existing state, etc. Please run
│ "terraform init" with either the "-reconfigure" or "-migrate-state" flags to
│ use the current configuration.
│
│ If the change reason above is incorrect, please verify your configuration
│ hasn't changed and try again. At this point, no changes to your existing
│ configuration or state have been made.

To initialize and migrate your current state file you can use the below command:

terraform init -migrate-state

Initializing the backend...
Do you want to copy existing state to the new backend?
  Pre-existing state was found while migrating the previous "local" backend to the
  newly configured "s3" backend. No existing state was found in the newly
  configured "s3" backend. Do you want to copy this state to the new "s3"
  backend? Enter "yes" to copy and "no" to start with an empty state.

  Enter a value: yes

Successfully configured the backend "s3"! Terraform will automatically
use this backend unless the backend configuration changes.
Initializing provider plugins...
- Reusing previous version of hashicorp/aws from the dependency lock file
- Using previously-installed hashicorp/aws v5.61.0

Terraform has been successfully initialized!

You may now begin working with Terraform. Try running "terraform plan" to see
any changes that are required for your infrastructure. All Terraform commands
should now work.

If you ever set or change modules or backend configuration for Terraform,
rerun this command to reinitialize your working directory. If you forget, other
commands will detect it and remind you to do so if necessary.

As you can see in the output after initializing the backend the terraform state is now stored in the S3 bucket, we can confirm that by going to the S3 bucket that we created and check under the directory path that we provided to see the state file stored there. yavarin.tech

Now let’s get back to our developers and see how these changes will effect their work flow.

New workflow

Developer A and B will pull the latest changes from the repository. As you might’ve guessed it now there are no terraform.tfstate and the terraform.tfstate.backup files. Developers can read the current state from the S3 bucket and their changes will be applied immediately to the state file in the S3 bucket. When developers add any new resources to the code they will try to see what effect does it have on the infrastructure or they just want to test if their code is working so they’d execute terraform plan. Terraform will automatically checks the changes against the state file and lists the changes that the new code will introduce. If the any of the developers are happy with their changes they have to use terraform apply to apply the changes to both the infrastructure and the state file. Let’s say developer A is first to apply his changes, and developer B is also ready to apply his changes. developer B will get the below prompt saying that the lock on state file can not be acquired.

Acquiring state lock. This may take a few moments...
╷
│ Error: Error acquiring the state lock
│
│ Error message: operation error DynamoDB: PutItem, https response error StatusCode: 400, RequestID:
│ CIGA2SV9O1FBBTOIPKVEC61UQRVV4KQNSO5AEMVJF66Q9ASUAAJG, ConditionalCheckFailedException: The conditional request failed
│ Lock Info:
│   ID:        8222d707-0ce7-3921-d128-6b01cc865b52
│   Path:      devopsprojects-terraformstate/devops-projects/us-east-1/terraform.tfstate
│   Operation: OperationTypeApply
│   Who:       developerA
│   Version:   1.9.2
│   Created:   2024-08-05 11:32:18.625884 +0000 UTC
│   Info:
│
│
│ Terraform acquires a state lock to protect the state from being written
│ by multiple users at the same time. Please resolve the issue above and try
│ again. For most commands, you can disable locking with the "-lock=false"
│ flag, but this is not recommended.

What happened behind the seen?

When developer A applied his changes Terraform created a record in the DynamoDB table with some information about the lock and as long as he is holding the lock no one can apply any changes. When developer B tried to apply terraform failed to acquire the lock of the state file and throw error about the situation. As you can see the error message is stating who is holding the lock, this way you can prevent simultaneous modification of the infrastructure to prevent interruption and missing resources. Now developer B has to wait for developer A to apply his changes and then developer B can acquire the lock and apply his changes.

Considerations and Conclusion

You need to note that suing a remote backend for terraform is not always suitable for you scenario. For instance if you are deploying the infrastructure using a CI/CD tool you might not be able to benefit from remote backends to the fullest. Also, you need keep in mind that migrating to and from the remote backend requires changes in your root module. So if for example you need to manually edit the state file (eg. importing previously created resources) you probably end up migrating from remote backends to local and apply your modifications then move back to the remote backend. As mentioned in the previous post with multiple people working on the same code and infrastructure you need a good workflow that allows people to work collaboratively and unblock them selves while doing so. If you are interested to read more about the configuration of a remote backend and what are other possibilities and option I re command reading the documents here.