Best Practices for Configuring Terraform Remote State Backend | Secure & Scalable Storage

Quick answer: A Terraform remote state backend stores the terraform.tfstate file in shared, durable storage — an S3 bucket, Azure Blob container, Google Cloud Storage bucket or HCP Terraform — instead of on one engineer’s laptop. The best-practice setup is versioned, encrypted storage with state locking, a separate state file per environment, and access restricted through IAM so only the pipeline and a few humans can read it.

State is the single most important file Terraform produces: lose it and Terraform forgets what it manages; corrupt it and your next apply can destroy production. This guide explains why remote state matters, how to configure S3, Azure and GCS backends correctly in Terraform 1.x, how locking and workspaces fit in, and the mistakes that cause real outages.

What Terraform state is and why local state fails

Every time you run terraform apply, Terraform records the mapping between your configuration and real resource IDs in a JSON state file. On the next run it compares that file with the live infrastructure to decide what to change. By default the file sits in your working directory as terraform.tfstate.

That works for a weekend project and breaks the moment a second person joins:

• Collaboration – two engineers with two copies of the state will each try to create the same resources, and one will fail or duplicate.
• Safety – a laptop dies or a git clean wipes the folder and the state is gone forever.
• Security – state contains sensitive values in plain text (database passwords, private keys generated by tls_private_key). Committing it to Git is a leak.
• Automation – a CI/CD runner is a fresh machine every time; it needs to fetch state from somewhere.

A remote backend fixes all four. If you are new to how Terraform tracks resources at all, the walkthrough in Creating and Managing Infrastructure Resources in Terraform shows the state file growing with each resource you add.

Choosing a backend

The rule of thumb is simple: use the object storage of the cloud you are already deploying to. It is cheap, extremely durable, and your access controls are already in place.

Hands-on: configuring the S3 backend properly

The backend block lives inside the terraform block. The bucket itself must exist before you run terraform init — a classic chicken-and-egg problem, solved by creating it once by hand, with a small bootstrap configuration that uses local state, or with the CLI.

# One-time bootstrap (ap-south-1 = Mumbai)
aws s3api create-bucket --bucket tkh-terraform-state-prod \
  --region ap-south-1 \
  --create-bucket-configuration LocationConstraint=ap-south-1

aws s3api put-bucket-versioning --bucket tkh-terraform-state-prod \
  --versioning-configuration Status=Enabled

aws s3api put-public-access-block --bucket tkh-terraform-state-prod \
  --public-access-block-configuration \
  BlockPublicAcls=true,IgnorePublicAcls=true,BlockPublicPolicy=true,RestrictPublicBuckets=true

Now the backend configuration. Terraform 1.10 added native S3 locking via a .tflock object, so a DynamoDB table is no longer required for new projects:

# backend.tf
terraform {
  required_version = ">= 1.10.0"

  backend "s3" {
    bucket       = "tkh-terraform-state-prod"
    key          = "networking/vpc/terraform.tfstate"
    region       = "ap-south-1"
    encrypt      = true
    use_lockfile = true          # native S3 locking, replaces DynamoDB
  }
}

Three details matter here. encrypt = true enables server-side encryption at rest; combine it with a kms_key_id if your compliance team requires customer-managed keys. The key is a path inside the bucket — use a hierarchy like <layer>/<component>/terraform.tfstate so one bucket can hold dozens of independent states. And versioning on the bucket gives you a free undo button: every apply creates a new object version you can restore if something goes wrong.

If you are on Terraform 1.9 or earlier, replace use_lockfile with dynamodb_table = "tkh-terraform-locks" and create a table with a string partition key named LockID.

Azure and Google Cloud equivalents

# Azure Blob Storage
terraform {
  backend "azurerm" {
    resource_group_name  = "rg-terraform-state"
    storage_account_name = "tkhtfstateprod"      # must be globally unique, lowercase
    container_name       = "tfstate"
    key                  = "networking/vnet.tfstate"
    use_azuread_auth     = true                  # no storage keys in config
  }
}

# Google Cloud Storage
terraform {
  backend "gcs" {
    bucket = "tkh-terraform-state-prod"
    prefix = "networking/vpc"                    # Terraform appends default.tfstate
  }
}

Both backends lock automatically, and both buckets should have versioning (Azure calls it blob versioning or soft delete) turned on. For Azure, prefer use_azuread_auth over storage account keys so access is governed by Entra ID roles rather than a shared secret.

Initialising, migrating and reading remote state

Run terraform init after adding the backend block. If a local state file exists, Terraform asks whether to copy it to the new backend — answer yes, confirm the remote copy exists, then delete the local file. To move between backends later, change the block and run terraform init -migrate-state.

Keep credentials and environment-specific values out of the backend block by using partial configuration. The block holds only what is common, and the pipeline supplies the rest:

# dev.s3.tfbackend
bucket = "tkh-terraform-state-dev"
key    = "networking/vpc/terraform.tfstate"
region = "ap-south-1"

# In CI
terraform init -backend-config=dev.s3.tfbackend

Other configurations can then read outputs from this state using the terraform_remote_state data source — for example, an application stack looking up the VPC ID produced by the networking stack. Remember that the reader needs read access to the bucket and can see everything in the state, so many teams publish a few values to SSM Parameter Store instead.

Seven remote state mistakes that cause outages

1. One giant state for everything. A single state file for networking, databases and twenty applications means every plan takes minutes and every mistake has a huge blast radius. Split by environment and by component.
2. No versioning on the bucket. Without object versions there is no rollback when a bad apply or a manual terraform state rm corrupts the file.
3. Skipping locking. Two pipelines applying at once will race, and the loser’s changes are written over. Always enable use_lockfile, a DynamoDB table, or use a backend with built-in locks.
4. Broad read access. State contains secrets. Grant s3:GetObject only to the CI role and a small admin group, and enable bucket access logging.
5. Using variables in the backend block. The backend is evaluated before variables exist, so bucket = var.state_bucket fails. Use partial configuration files instead, as shown above. The limits on where variables work are covered in Using Variables and Expressions in Terraform.
6. Storing the state bucket in the same state it holds. If the bucket is managed by the configuration whose state lives in it, destroying that configuration deletes its own state. Bootstrap the bucket separately.
7. Forgetting terraform force-unlock etiquette. A crashed pipeline can leave a stale lock. Confirm nobody else is running before force-unlocking with the exact lock ID shown in the error.

Frequently asked questions

Should I use workspaces or separate state keys for dev, staging and prod?

Workspaces share one backend configuration and one bucket, which makes it easy to accidentally apply dev changes to prod. Most teams use separate backend config files (and often separate buckets or accounts) per environment and reserve workspaces for short-lived feature branches.

Is state encrypted by default?

Not by Terraform itself. S3 now encrypts new objects with SSE-S3 by default, but you should still set encrypt = true and consider a KMS key. Azure and GCS encrypt at rest by default. Encryption in transit is always HTTPS.

Can I still use DynamoDB for locking?

Yes. The dynamodb_table argument remains supported and is required on Terraform versions older than 1.10. New projects should prefer use_lockfile = true to avoid managing an extra table.

What is the difference between the remote and cloud backends?

Both talk to HCP Terraform. The cloud block is the newer, recommended syntax and supports tag-based workspace selection; remote is the legacy form kept for compatibility.

Key takeaways

• Remote state is mandatory the moment a second person or a pipeline touches your Terraform code.
• Use the object storage of your cloud with versioning, encryption and locking enabled.
• Split state per environment and component; keep the bucket itself outside the state it stores.
• Lock down read access — state is a secrets file, not just metadata.

Want to design the state, module and pipeline structure that real platform teams use across AWS and Azure? Our DevOps course covers Terraform from first init to multi-account production setups, with mentor support and placement assistance. For hands-on demos, visit our YouTube channel.