Terraform is powerful because it allows infrastructure to be managed as code. But behind every Terraform project, there is one very important file that many beginners do not fully understand at first: the Terraform state file.

In this assignment, I built a practical Terraform State Recovery project on AWS. The goal was to simulate accidental state deletion, recover existing infrastructure using terraform import, and detect drift when someone makes manual changes outside Terraform.

This is an important DevOps skill because real-world infrastructure work is not only about creating resources. It is also about recovering from mistakes, troubleshooting state issues, and keeping infrastructure aligned with code.

Project Objective

The objective of this assignment was to simulate a realistic Terraform recovery scenario.

The project covered:

Accidental deletion of Terraform state
Recovery using terraform import
Drift detection using terraform plan
Drift correction
Remote backend best practices
S3 state storage
DynamoDB state locking

The infrastructure used for this demo was simple but practical:

AWS VPC
Public subnet
Security group

The focus was not on building a large architecture. The focus was on understanding how Terraform remembers infrastructure and how to recover that memory if it is lost.

What is Terraform State?

Terraform state is Terraform’s memory.

When Terraform creates a resource in AWS, it records that resource in a state file. For example, if Terraform creates a VPC, subnet, or security group, Terraform stores the real AWS resource IDs inside the state file.

Without state, Terraform does not know what it already created.

For example, if AWS still has a VPC but Terraform state is missing, Terraform may think the VPC does not exist and try to create it again.

That is why Terraform state is one of the most important parts of Infrastructure as Code.

Why State Recovery Matters

In real DevOps work, state issues can happen because of:

Accidental deletion
Incorrect backend migration
Manual file removal
Broken CI/CD pipeline
Wrong workspace or backend
S3 object deletion
Team members running Terraform from different places

If state is lost, the infrastructure may still exist in AWS, but Terraform no longer tracks it.

This creates a dangerous situation:

AWS resources exist
Terraform does not know about them
Future plans may try to recreate or change resources incorrectly

State recovery is the process of reconnecting Terraform code with existing infrastructure.

Project Setup

The project started with a new Git repository:

mkdir Asignment-4
cd Asignment-4
git init

The folder structure was simple:

Asignment-4/
└── environments/
    └── dev/
        ├── provider.tf
        ├── main.tf
        ├── variables.tf
        ├── terraform.tfvars
        ├── outputs.tf
        └── recovery-notes.md

This structure keeps the dev environment separate and makes the project easy to understand.

Terraform Provider Configuration

The provider file defines Terraform version requirements and the AWS provider.

terraform {
  required_version = ">= 1.6.0"

  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 6.0"
    }
  }
}

provider "aws" {
  region = var.aws_region

  default_tags {
    tags = var.common_tags
  }
}

This configuration tells Terraform to use AWS as the cloud provider and to apply common tags to supported resources.

The use of variables keeps the code flexible. Instead of hardcoding the region, the region is provided through terraform.tfvars.

Infrastructure Code

The project creates three AWS resources:

VPC
Public subnet
Security group

Example VPC resource:

resource "aws_vpc" "recovery_demo" {
  cidr_block           = var.vpc_cidr
  enable_dns_support   = true
  enable_dns_hostnames = true

  tags = {
    Name = "${var.environment}-state-recovery-vpc"
  }
}

This creates a VPC with DNS support and a clear name tag.

The subnet is created inside the VPC:

resource "aws_subnet" "public" {
  vpc_id                  = aws_vpc.recovery_demo.id
  cidr_block              = var.public_subnet_cidr
  availability_zone       = var.availability_zone
  map_public_ip_on_launch = true

  tags = {
    Name = "${var.environment}-state-recovery-public-subnet"
  }
}

The important line here is:

vpc_id = aws_vpc.recovery_demo.id

This tells Terraform that the subnet belongs to the VPC created above.

The security group allows HTTP traffic for demo purposes:

resource "aws_security_group" "web_sg" {
  name        = "${var.environment}-state-recovery-web-sg"
  description = "Security group for Terraform state recovery demo"
  vpc_id      = aws_vpc.recovery_demo.id

  ingress {
    description = "Allow HTTP traffic for demo"
    from_port   = 80
    to_port     = 80
    protocol    = "tcp"
    cidr_blocks = var.http_allowed_cidrs
  }

  egress {
    description = "Allow all outbound traffic"
    from_port   = 0
    to_port     = 0
    protocol    = "-1"
    cidr_blocks = ["0.0.0.0/0"]
  }

  tags = {
    Name = "${var.environment}-state-recovery-web-sg"
  }
}

This security group is used to demonstrate how Terraform tracks AWS resources in state.

Variable Values

The actual values are stored in terraform.tfvars.

aws_region         = "us-east-1"
environment        = "dev"
vpc_cidr           = "10.60.0.0/16"
public_subnet_cidr = "10.60.1.0/24"
availability_zone  = "us-east-1a"
http_allowed_cidrs = ["0.0.0.0/0"]

common_tags = {
  Project     = "Asignment-4"
  Environment = "dev"
  Owner       = "Hashim."
  ManagedBy   = "Terraform"
  Purpose     = "StateRecoveryDemo"
}

This keeps environment-specific settings outside the main Terraform logic.

Deploying the Infrastructure

The infrastructure was deployed using the normal Terraform workflow:

cd environments/dev
terraform init
terraform fmt
terraform validate
terraform plan
terraform apply

After applying, Terraform created the VPC, subnet, and security group in AWS.

Terraform also created a local state file:

terraform.tfstate

This state file contained the mapping between the Terraform code and real AWS resources.

To confirm what Terraform was tracking, I used:

terraform state list

Expected output:

aws_security_group.web_sg
aws_subnet.public
aws_vpc.recovery_demo

This confirmed that Terraform knew about all three resources.

Simulating Accidental State Deletion

To simulate state loss, I first created a backup:

cp terraform.tfstate terraform.tfstate.backup

Then I deleted the state file:

rm terraform.tfstate

This simulates a real-world mistake where the Terraform state file is accidentally deleted.

At this point, the AWS resources still exist.

But Terraform no longer knows about them.

Running:

terraform state list

would show that Terraform has no tracked resources.

This is the key lesson:

Deleting state does not delete AWS infrastructure.
It only deletes Terraform’s memory of that infrastructure.

What Happens After State Loss?

After deleting the state file, running:

terraform plan

causes Terraform to behave as if the resources do not exist.

Terraform may try to create the VPC, subnet, and security group again.

This is dangerous because the real infrastructure already exists in AWS.

The correct solution is not to recreate everything.

The correct solution is to import the existing resources back into Terraform state.

Finding Existing AWS Resource IDs

Before importing, Terraform needs the real AWS IDs.

I used AWS CLI to find them.

Find the VPC ID:

aws ec2 describe-vpcs \
  --filters "Name=tag:Name,Values=dev-state-recovery-vpc" \
  --query "Vpcs[0].VpcId" \
  --output text

Find the subnet ID:

aws ec2 describe-subnets \
  --filters "Name=tag:Name,Values=dev-state-recovery-public-subnet" \
  --query "Subnets[0].SubnetId" \
  --output text

Find the security group ID:

aws ec2 describe-security-groups \
  --filters "Name=group-name,Values=dev-state-recovery-web-sg" \
  --query "SecurityGroups[0].GroupId" \
  --output text

These commands return IDs like:

vpc-xxxxxxxx
subnet-xxxxxxxx
sg-xxxxxxxx

These IDs are required for import.

Recovering State Using terraform import

The terraform import command reconnects existing AWS resources with Terraform state.

Import the VPC:

terraform import aws_vpc.recovery_demo vpc-xxxxxxxx

Import the subnet:

terraform import aws_subnet.public subnet-xxxxxxxx

Import the security group:

terraform import aws_security_group.web_sg sg-xxxxxxxx

This does not create new infrastructure.

It only updates Terraform state and tells Terraform:

This existing AWS resource belongs to this Terraform resource block.

For example:

terraform import aws_vpc.recovery_demo vpc-xxxxxxxx

means:

The existing AWS VPC with this ID should now be tracked as aws_vpc.recovery_demo.

Verifying Recovery

After importing all resources, I checked state again:

terraform state list

Expected output:

aws_security_group.web_sg
aws_subnet.public
aws_vpc.recovery_demo

Then I ran:

terraform plan

The ideal result is:

No changes. Your infrastructure matches the configuration.

If Terraform still shows changes, that means the imported AWS resources do not fully match the Terraform code.

This can happen if some resource settings were created differently or changed manually.

Drift Detection

Drift happens when the real infrastructure is different from the Terraform code.

For example:

Terraform code says VPC Name tag = dev-state-recovery-vpc
AWS Console manually changes it to dev-state-recovery-vpc-manual-change

Now Terraform code and AWS are not aligned.

To detect drift, run:

terraform plan

Terraform will compare:

Terraform code
Terraform state
Actual AWS infrastructure

If it finds differences, it shows what needs to change.

This is called drift detection.

Handling Drift

There are two ways to handle drift.

The first option is to keep Terraform as the source of truth.

If the manual AWS change was wrong, run:

terraform apply

Terraform will revert AWS back to what the code says.

The second option is to accept the manual change.

If the manual change was correct, update the Terraform code to match the new desired state, then run:

terraform plan
terraform apply

The important rule is simple:

Do not leave AWS and Terraform code different from each other.

Either fix AWS or update the code.

Modern Best Practice: Remote Backend

Using local state is risky.

For real projects, state should be stored remotely in S3 with locking.

A modern backend setup includes:

S3 bucket for remote state
Bucket versioning for recovery
Encryption for protection
DynamoDB locking to prevent simultaneous state changes

Example backend:

terraform {
  backend "s3" {
    bucket         = "asignment-4-terraform-state-34-238-244-58"
    key            = "dev/terraform.tfstate"
    region         = "us-east-1"
    dynamodb_table = "asignment-4-terraform-locks"
    encrypt        = true
  }
}

This improves safety because state is no longer stored only on one machine.

S3 versioning also helps recover previous versions if the state object is accidentally changed or deleted.

Lessons Learned

This assignment taught me that Terraform state is not just a background file. It is a critical part of infrastructure management.

The most important lessons were:

Terraform state is Terraform’s memory
Deleting state does not delete infrastructure
State loss can cause Terraform to recreate resources
terraform import can rebuild state tracking
terraform plan can detect drift
Remote state is safer than local state
Manual AWS Console changes should be avoided

Why This Project Matters for DevOps

This project is valuable because state recovery is a real operational skill.

In DevOps, Cloud Engineering, Platform Engineering, and SRE roles, engineers must know how to recover from mistakes safely.

Creating infrastructure is only one part of the job.

Managing state, detecting drift, recovering resources, and avoiding downtime are equally important.

This assignment demonstrates practical understanding of:

Terraform state management
Disaster recovery
AWS CLI usage
Resource import
Drift detection
Infrastructure troubleshooting
Remote backend best practices

Conclusion

Terraform makes infrastructure repeatable and manageable, but Terraform depends heavily on state.

If the state is lost, Terraform loses track of the infrastructure it manages. The infrastructure may still exist in AWS, but Terraform no longer knows how to control it.

By using terraform import, we can reconnect existing resources to Terraform state. By using terraform plan, we can detect drift. By using remote backend storage, versioning, and locking, we can reduce the risk of state loss in the first place.

This assignment shows an important real-world DevOps lesson:

Good infrastructure engineering is not only about building systems.
It is also about recovering them safely when something goes wrong.