Destroying AWS Infrastructure

This guide covers the generated GitHub Actions workflow, .github/workflows/terraform-deploy.yml. Generated repositories also contain a .gitlab-ci.yml; GitLab CI teardown is not covered here.

The workflow clears database deletion guards automatically, and its Kubernetes destroy removes the AWS resources that Kubernetes controllers created. After teardown, check AWS for leftovers anyway: an empty Terraform state does not guarantee that every resource has been removed.

1. Normal teardown

These instructions apply to generated repositories whose .github/workflows/terraform-deploy.yml includes Relax database teardown guards. Older generated repositories may lack this step; inspect your workflow before following these instructions.

1. In the generated repository, open Actions, select the Terraform deployment workflow, and choose Run workflow on the branch containing the deployed configuration.
2. Set terraform_action to destroy.
3. Choose the skip_final_snapshot setting:
   • Unchecked (default): take final RDS/Aurora snapshots. Retained snapshots can incur storage charges and must be managed separately after teardown.
   • Checked: delete without a final database snapshot. Data absent from an existing usable backup cannot be recovered. The input is labelled Destroy without taking a final DB snapshot (data is unrecoverable).
4. Start the workflow. It destroys the Kubernetes stack first, then the AWS stack, and waits for approval on a GitHub issue before each one:
   • destroy:k8s opens an issue titled Approve Terraform destroy (Kubernetes) — run …. Only after the Kubernetes destroy succeeds does destroy:aws open Approve Terraform destroy (AWS) — run …, which also states the snapshot policy you chose.
   • To approve, comment approve, approved, lgtm or yes. To stop, comment reject, rejected, deny, denied or no, or close the issue.
   • The commenter needs Write, Maintain or Admin access to the repository. The person who started the workflow may approve it.
   • Each request expires after 55 minutes. A rejected or expired request fails that job before Terraform runs.
   • Issues must be enabled on the repository, or the workflow cannot open the request.

If your workflow has no Authorize Kubernetes destruction step, it was generated before approvals were added and starts destroying as soon as you run it.

Before AWS destruction, Relax database teardown guards applies deletion protection and snapshot settings for you. You do not need to edit terraform.auto.tfvars or pass database variables manually for normal teardown. The RDS module derives its snapshot name; the workflow supplies Aurora with aurora-final-${RUN_ID}, using the GitHub Actions run ID.

If the relaxation step fails, Terraform Destroy is skipped. Use section 3 instead of repeatedly rerunning the same failed workflow.

2. Checking for leftover resources

Controllers create AWS resources outside Terraform state. The workflow's Kubernetes destroy deletes the ArgoCD applications first and waits for them to be removed while the controllers are still running, so the controllers delete the load balancers, security groups and instances they created. If that cleanup does not finish, those resources stay behind after the cluster is gone. Before teardown, record the cluster name, AWS account, region, VPC ID, and resource IDs so you can find leftovers even after EKS has been deleted.

Resource	How to find it	Why it matters
Controller-created NLBs/ALBs, including ingress-nginx load balancers	Kubernetes Services of type LoadBalancer and Ingress objects; EC2 Load Balancers, filtered by VPC, then inspect cluster/service/ingress tags	Load balancers can remain billable and retain network dependencies
Controller-created k8s-* security groups	EC2 Security Groups, filtered by VPC and group-name prefix; inspect tags, attached network interfaces, and referencing rules	Remaining groups and dependencies can block VPC deletion
Karpenter-provisioned EC2 instances	Kubernetes NodeClaims; EC2 Instances, filtered by VPC and tags such as karpenter.sh/nodepool and kubernetes.io/cluster/<cluster-name>	Karpenter launches instances through the fleet API outside Terraform state; instances may keep running after cluster deletion

Tags vary with controller versions. A k8s-* name or Karpenter tag is a discovery hint, not sufficient proof that a resource belongs to this environment. Confirm ownership using the recorded VPC, cluster tags, and Kubernetes object names; do not bulk-delete resources by prefix alone.

If the Kubernetes destroy fails

When destroy:k8s fails, the workflow does not start destroy:aws, so the cluster still exists. Clean up from the cluster before you run the workflow again.

Connect with the AWS credentials stored in the repository secrets (AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY). The generated cluster grants administrator access only to the identity that created it, which is the workflow's identity; any other identity gets Unauthorized unless it has an EKS access entry. The cluster is named <global prefix>eks-<environment>, which is also the eks_cluster_name Terraform output.

Configure access, confirm the Kubernetes context and inspect controller-owned objects:

aws eks update-kubeconfig --name CLUSTER_NAME --region REGION
kubectl config current-context
kubectl get services --all-namespaces --field-selector spec.type=LoadBalancer
kubectl get ingress --all-namespaces
kubectl get nodeclaims -o wide

The NodeClaims command applies only when Karpenter is installed.

1. Stop workload reconciliation that would recreate the resources being removed. Delete the environment's Ingress objects and LoadBalancer Services while their controllers are still running. For example, after substituting the confirmed names:

   kubectl delete ingress INGRESS_NAME -n NAMESPACE
   kubectl delete service SERVICE_NAME -n NAMESPACE

   Wait for their AWS load balancers to disappear before removing the controllers. AWS also calls for this ordering in its EKS cluster deletion instructions.

2. Stop or scale down workloads that would trigger replacement Karpenter capacity. Keep the Karpenter controller running on capacity that will remain available during cleanup. Delete each confirmed NodeClaim:

   kubectl delete nodeclaim NODECLAIM_NAME

   Wait for the corresponding EC2 instances to terminate. Karpenter's termination controller uses finalizers to drain nodes and terminate instances; do not remove those finalizers simply to force the Kubernetes objects to disappear.

3. Check for remaining controller-created security groups. Allow controller cleanup to finish, then investigate attached interfaces and referencing rules before removing confirmed orphan groups. Leave Terraform-managed groups to Terraform, and leave groups still used by live EKS resources in place.

4. Run the workflow with destroy again, then perform the checks below.

After teardown, or if the cluster is already gone

Use the recorded account, region, VPC, and tags in the AWS console:

• Delete confirmed orphan load balancers from EC2 Load Balancers, and inspect their target groups for leftovers.
• Terminate confirmed orphan Karpenter instances from EC2 Instances. Stopping an instance is not complete cleanup. Inspect any retained EBS volumes separately.
• After load balancer and instance cleanup, inspect remaining network interfaces and security group references. Remove confirmed orphan groups once their dependencies are gone. For service-managed interfaces, remove the owning resource rather than trying to force-delete the interface.
• If Terraform failed on VPC deletion, rerun a reviewed destroy after resolving these dependencies. Follow section 3 if workflow guard relaxation also fails.

Repeat the inventory until no unintended load balancers, non-terminated Karpenter instances, or orphan network dependencies remain. Confirm that the intended VPC was deleted and that any final snapshots you chose to retain exist. Retained snapshots and volumes can still incur charges after compute resources are gone.

3. When the guard-relaxation step fails

The relaxation step is an untargeted terraform apply. On a partially applied environment, it can attempt to create infrastructure that an earlier apply did not finish. If that attempt fails, the AWS destroy step never runs. This is a known limitation of the generated workflow; the manual recovery below works around it without changing the workflow.

Prepare the existing state and snapshot settings

Use a checkout of the generated repository and the same Terraform version, provider lock file, AWS account/region, credentials, input variables, and backend as the failed run. Ensure no other apply or destroy is running. Confirm the Kubernetes stack is destroyed first. If its Terraform stack still contains resources, review and destroy that stack from terraform/kubernetes before destroying AWS. Do not create a new backend or discard state to bypass an error.

From the generated repository root:

cd terraform/aws
terraform init -reconfigure
terraform workspace show
terraform state list

Confirm that the configured backend and selected workspace contain the intended environment. Supply any workflow-only secrets or variables through your normal secure mechanism; the checkout alone may not contain all required inputs.

Use these database settings during recovery:

Variable	Destroy value	Purpose
rds_deletion_protection	false	Allows RDS deletion
aurora_deletion_protection	false	Allows Aurora deletion
rds_apply_immediately	true	Applies the RDS change before destroy
aurora_apply_immediately	true	Already defaults to true, but must remain enabled

Choose final-snapshot behavior explicitly:

Variable setting	Behavior
rds_skip_final_snapshot = false	Creates a final RDS snapshot
rds_skip_final_snapshot = true	Deletes RDS without a final snapshot
aurora_skip_final_snapshot = false	Requires a valid final snapshot identifier
aurora_skip_final_snapshot = true	Deletes Aurora without a final snapshot

For manual recovery, create a local teardown.tfvars in terraform/aws with the following contents. Keep false to retain final snapshots; set both skip values to true only if you intend to omit them. Replace the example Aurora snapshot identifier with a unique name for this recovery attempt when Aurora is enabled. Do not commit the recovery file or saved plans.

rds_deletion_protection          = false
rds_apply_immediately            = true
rds_skip_final_snapshot          = false
aurora_deletion_protection       = false
aurora_apply_immediately         = true
aurora_skip_final_snapshot       = false
aurora_final_snapshot_identifier = "aurora-final-recovery-20260916-001"

Apply only the required database changes, then destroy

Find the exact existing aws_db_instance and/or aws_rds_cluster resource addresses in terraform state list. Use terraform state show 'ADDRESS' to confirm each database identifier. Target the existing database resources rather than an entire module; do not target databases that were never created.

For each required address, add a quoted -target argument to this plan command, replacing the placeholder below:

terraform plan -var-file=teardown.tfvars \
  -target='EXISTING_DATABASE_RESOURCE_ADDRESS' \
  -out=teardown-guards.tfplan
terraform show teardown-guards.tfplan

Review the plan before continuing. It should update the intended database guards and snapshot settings without creating, replacing, or deleting infrastructure. Terraform targeting includes dependencies, so it can still pull in unfinished resources. If it does, or planning fails, stop and have the environment maintainer resolve that specific dependency with the failed run and resource addresses. This is not a universal bypass for every partially applied configuration.

Apply only an acceptable saved plan:

terraform apply teardown-guards.tfplan

Confirm deletion protection is disabled in AWS and inspect the database state with terraform state show to verify the selected snapshot settings, including Aurora's identifier. Snapshot options must be recorded before deletion; simply passing different values to a destroy command is not a substitute for applying them. If no databases were created, skip the database apply entirely.

Next, create a full destroy plan in the same directory:

terraform plan -destroy -var-file=teardown.tfvars -out=teardown-destroy.tfplan
terraform show teardown-destroy.tfplan

After reviewing the resources to be deleted, execute that saved plan:

terraform apply teardown-destroy.tfplan

This runs the reviewed destroy directly, avoiding the workflow's untargeted relaxation apply. Rerunning the workflow instead would execute that step again. If deletion fails, address the reported dependency, then generate and review a new destroy plan. Finish with the AWS inventory checks in section 2, and remove the local recovery file and saved plans when recovery is complete.