Opentofu on Citizix, The Best blog for Linux, Opensource, Devops, Cloud and all things awesome tech.

Create AWS VPC Peering in the Same Account with Terraform

Mon, 27 Jun 2022 06:06:32 +0000

Amazon VPC peering enables a network connection between two private VPCs so traffic can be routed from one VPC to the other using private IP addresses. You can peer your VPC with another VPC in the same region, in a different region, or in a different AWS account.

AWS creates the peering connection using the existing infrastructure of the VPC. A VPC peering connection is not a gateway or a VPN connection, and there is no single point of failure or bandwidth bottleneck. It makes it easy to transfer data from VPC to VPC.

This guide covers the simplest case: both VPCs are in the same AWS account and the same region, so Terraform can create and accept the request in one step. It assumes the VPCs already exist. If you need help creating one, check out this guide: Terraform AWS VPC with Public and Private subnets with NAT.

We will:

Look up both VPCs by their Name tag.
Create the peering connection and accept it automatically.
Enable DNS resolution across the peering.
Add a route to the other VPC’s CIDR in every route table on both sides.
Allow ICMP in the security groups and test with ping.

Prerequisites

Terraform 1.6 or later, or OpenTofu. Every command below works with tofu in place of terraform.
AWS credentials for the account that owns both VPCs, with permission to manage VPC peering, routes and security groups.
Two VPCs in the same region with non-overlapping CIDR blocks, for example 10.10.0.0/16 and 10.20.0.0/16. AWS refuses to peer VPCs whose CIDRs overlap.
For the DNS part: enableDnsSupport and enableDnsHostnames turned on in both VPCs (the defaults for VPCs created by the console wizard and most Terraform modules).

Note that peering is not transitive: if VPC A is peered with B and B with C, A still cannot reach C. Each pair needs its own peering connection, or use a Transit Gateway once you have more than a handful of VPCs.

Provider and variables

Pin the AWS provider. This guide uses the current major version, 6.x. Save this as versions.tf:

terraform {
 required_version = ">= 1.6"

 required_providers {
 aws = {
 source = "hashicorp/aws"
 version = "~> 6.0"
 }
 }
}

provider "aws" {
 region = var.region
}

Let us define the variables we will use. Save the following as vars.tf:

variable "region" {
 type = string
 description = "AWS region both VPCs live in"
 default = "eu-west-1"
}

variable "name" {
 type = string
 description = "Name of the peering connection"
}

variable "requester_vpc_name" {
 type = string
 description = "Name tag of the requester VPC"
}

variable "accepter_vpc_name" {
 type = string
 description = "Name tag of the accepter VPC"
}

And give them values in terraform.tfvars:

name = "dev-to-shared"
requester_vpc_name = "dev-vpc"
accepter_vpc_name = "shared-vpc"

Look up the VPCs

Next, using the given VPC names, query both VPCs. Start main.tf with:

data "aws_vpc" "requester" {
 filter {
 name = "tag:Name"
 values = [var.requester_vpc_name]
 }
}

data "aws_vpc" "accepter" {
 filter {
 name = "tag:Name"
 values = [var.accepter_vpc_name]
 }
}

The data sources give us each VPC’s id and cidr_block, so there is no need to pass the VPC IDs in as separate variables.

Create the peering connection

Add the peering connection to main.tf:

resource "aws_vpc_peering_connection" "this" {
 vpc_id = data.aws_vpc.requester.id
 peer_vpc_id = data.aws_vpc.accepter.id
 auto_accept = true

 requester {
 allow_remote_vpc_dns_resolution = true
 }

 accepter {
 allow_remote_vpc_dns_resolution = true
 }

 tags = {
 Name = var.name
 }
}

auto_accept = true only works because both VPCs are in the same account and region. For cross-account or cross-region peering, you create the request here and accept it with an aws_vpc_peering_connection_accepter resource using a provider for the other account or region.
allow_remote_vpc_dns_resolution lets instances resolve the public DNS hostnames of instances in the other VPC to their private IPs. Without it, ec2-203-0-113-10.eu-west-1.compute.amazonaws.com resolves to the public IP and the traffic does not go over the peering.

Update the route tables on both sides

Now that the peering connection is created, we have to update the route table entries on both sides to send traffic via the peering connection. A single VPC can have multiple route tables (typically one public and one per private subnet tier or AZ), and I wanted the code to work irrespective of how many route tables each VPC has. So we look up all the route tables and loop over them with for_each, creating a route to the other VPC’s CIDR block via the peering connection.

data "aws_route_tables" "requester" {
 vpc_id = data.aws_vpc.requester.id
}

data "aws_route_tables" "accepter" {
 vpc_id = data.aws_vpc.accepter.id
}

# Requester route tables -> accepter CIDR
resource "aws_route" "requester_to_accepter" {
 for_each = toset(data.aws_route_tables.requester.ids)

 route_table_id = each.value
 destination_cidr_block = data.aws_vpc.accepter.cidr_block
 vpc_peering_connection_id = aws_vpc_peering_connection.this.id
}

# Accepter route tables -> requester CIDR
resource "aws_route" "accepter_to_requester" {
 for_each = toset(data.aws_route_tables.accepter.ids)

 route_table_id = each.value
 destination_cidr_block = data.aws_vpc.requester.cidr_block
 vpc_peering_connection_id = aws_vpc_peering_connection.this.id
}

for_each keys each route by its route table ID, so adding or removing a route table later only touches that one route. With count, removing a table from the middle of the list would shift the indexes and make Terraform recreate unrelated routes.

The routes send the whole peer VPC CIDR. It is also possible to route only specific subnets over the peering connection by using subnet CIDRs as destination_cidr_block.

After the apply, every route table on the requester side has an entry like this (and the mirror image on the accepter side):

Destination	Target
`10.10.0.0/16`	`local`
`10.20.0.0/16`	`pcx-0123456789abcdef0`

Allow traffic in the security groups

Routes only make the path exist. Security groups still decide what is allowed in. This example allows ICMP (ping) from each VPC into the other VPC’s default security group. Save it as security.tf, and add rules for the ports your workloads actually use (5432 for PostgreSQL, 443 for an internal API, and so on).

data "aws_security_group" "requester_default" {
 vpc_id = data.aws_vpc.requester.id
 name = "default"
}

data "aws_security_group" "accepter_default" {
 vpc_id = data.aws_vpc.accepter.id
 name = "default"
}

# Allow ping from the requester VPC into the accepter VPC
resource "aws_vpc_security_group_ingress_rule" "accepter_icmp_from_requester" {
 security_group_id = data.aws_security_group.accepter_default.id
 cidr_ipv4 = data.aws_vpc.requester.cidr_block
 ip_protocol = "icmp"
 from_port = -1
 to_port = -1
 description = "ICMP from peered VPC ${var.requester_vpc_name}"
}

# Allow ping from the accepter VPC into the requester VPC
resource "aws_vpc_security_group_ingress_rule" "requester_icmp_from_accepter" {
 security_group_id = data.aws_security_group.requester_default.id
 cidr_ipv4 = data.aws_vpc.accepter.cidr_block
 ip_protocol = "icmp"
 from_port = -1
 to_port = -1
 description = "ICMP from peered VPC ${var.accepter_vpc_name}"
}

aws_vpc_security_group_ingress_rule is the current way to manage individual rules. Avoid mixing it with inline ingress blocks or the older aws_security_group_rule on the same security group: they overwrite each other.

In the same region you can also reference the peer VPC’s security group ID instead of a CIDR (referenced_security_group_id), which is tighter than allowing a whole CIDR block.

Outputs

Save this as outputs.tf:

output "peering_connection_id" {
 value = aws_vpc_peering_connection.this.id
}

output "peering_status" {
 value = aws_vpc_peering_connection.this.accept_status
}

Plan and apply

terraform init
terraform fmt
terraform validate
terraform plan -out=peering.plan
terraform apply peering.plan

validate should print:

Success! The configuration is valid.

After the apply, peering_status should be active. Delete peering.plan once applied: saved plans can contain sensitive values and do not belong in Git.

Verify the peering

Check the connection status from the AWS CLI:

aws ec2 describe-vpc-peering-connections \
 --filters Name=tag:Name,Values=dev-to-shared \
 --query 'VpcPeeringConnections[].{Id:VpcPeeringConnectionId,Status:Status.Code}' \
 --output table

The status code should be active. pending-acceptance means the accept step did not happen; failed usually means overlapping CIDRs.

Check the routes landed in a requester route table:

aws ec2 describe-route-tables \
 --filters Name=vpc-id,Values=<requester-vpc-id> \
 --query 'RouteTables[].Routes[?VpcPeeringConnectionId!=`null`].[DestinationCidrBlock,VpcPeeringConnectionId,State]' \
 --output text

Then test from an instance. From an instance in the requester VPC, ping the private IP of an instance in the accepter VPC (both using the default security group from above):

ping -c 3 10.20.1.15

You should get three replies. With DNS resolution enabled, the other instance’s public DNS hostname resolves to its private IP from inside the peered VPC, so this should print 10.20.1.15 rather than the public address:

dig +short ec2-203-0-113-25.eu-west-1.compute.amazonaws.com

Troubleshooting

Ping times out: check, in order, the route tables on both sides (the reply needs a route back), the security group on the target, and any network ACLs on the subnets. ICMP is a common one to forget in NACLs.
Peering stuck in pending-acceptance: auto_accept only works within one account and region. Use aws_vpc_peering_connection_accepter otherwise.
Error: ... no matching EC2 VPC found: the Name tag in your variables does not match the VPC, or the provider is pointed at the wrong region.
Routes keep being recreated: you are probably mixing these aws_route resources with inline route blocks in an aws_route_table resource elsewhere. Pick one style per route table.

Conclusion

In this guide, we learnt how to create a VPC peering connection between two VPCs in the same AWS account and region, with routes in every route table on both sides, DNS resolution across the peering, security group rules, and a ping test to prove it works.

Create a VPC Network and GKE Cluster in GCP with Terraform

Mon, 23 May 2022 06:27:50 +0000

In this guide, we will build a Google Kubernetes Engine (GKE) cluster in Google Cloud Platform using Terraform. Terraform allows you to develop cloud infrastructure by automating repetitive tasks.

Creating a GKE cluster in the console can be tiring, especially if you have to create multiple clusters with different parameters such as node types, node sizes etc. Terraform was created to solve that problem. It allows you to have the instructions as code that can be used to plan, deploy, modify, and destroy the clusters programmatically.

We will create:

A custom VPC network with a public and a private subnet, with secondary ranges for Pods and Services (a VPC-native cluster).
A Cloud Router and Cloud NAT, so nodes without public IPs can still reach the internet.
A regional GKE Standard cluster with private nodes, a release channel and Workload Identity.
A separately managed node pool running as a dedicated, least-privilege service account.

We will also look at the Autopilot alternative, connect with kubectl, and go over what it costs.

Everything here works the same with OpenTofu: replace terraform with tofu.

Check out these:

Requirements

You need the following to proceed:

A Google Cloud project with billing enabled. GCP organizes resources into projects. Create one in the GCP console and make note of the project ID.

The Compute Engine and Kubernetes Engine APIs enabled for the project:

gcloud services enable compute.googleapis.com container.googleapis.com --project citizix-prj

Credentials Terraform can use. On your own machine, the simplest option is Application Default Credentials:
```
gcloud auth application-default login
```
In CI, use a service account. This guide covers it: Create a Terraform Service Account in GCP. Point Terraform at its key with export GOOGLE_APPLICATION_CREDENTIALS=/path/to/key.json, or better, use Workload Identity Federation so there is no long-lived key at all. The account needs at least roles/container.admin, roles/compute.networkAdmin, roles/iam.serviceAccountAdmin, roles/iam.serviceAccountUser and roles/resourcemanager.projectIamAdmin for everything below; roles/editor is too broad for anything but a throwaway project, and cannot grant IAM roles anyway.

Step 1 – Install Terraform, gcloud and kubectl

Terraform 1.6 or later from the official install page, or OpenTofu.
gcloud CLI from the Google Cloud SDK install guide.
kubectl and the GKE auth plugin. Since Kubernetes 1.26, kubectl cannot authenticate to GKE without gke-gcloud-auth-plugin. Install both through gcloud:
```
gcloud components install kubectl gke-gcloud-auth-plugin
```
If you installed gcloud from apt or dnf, install the google-cloud-cli-gke-gcloud-auth-plugin package instead. Check it works:
```
gke-gcloud-auth-plugin --version
```

Step 2 – Adding the project code

In this section we will create the files that will contain the code for our resources. First create a directory and switch to it:

mkdir gcp-gke
cd gcp-gke

Providers and locals

First we will have to specify the providers. Terraform relies on plugins called “providers” to interact with cloud providers, SaaS providers, and other APIs. Terraform configurations must declare which providers they require so that Terraform can install and use them.

This is main.tf, where we define the google provider and pin it to the current major version, 8.x. Provider majors change defaults and remove arguments, so ~> 8.0 lets you pick up fixes without an unplanned jump to 9.0. We are also defining some locals that we can reuse.

A local value assigns a name to an expression, so you can use the name multiple times within a module instead of repeating the expression. Local values are like a function’s temporary local variables.

locals {
 env = "dev"
 project = "citizix"
 region = "europe-west1"
 project_id = "citizix-prj"
}

terraform {
 required_version = ">= 1.6"

 required_providers {
 google = {
 source = "hashicorp/google"
 version = "~> 8.0"
 }
 }
}

provider "google" {
 project = local.project_id
 region = local.region
}

There is no credentials argument: the provider picks up Application Default Credentials or GOOGLE_APPLICATION_CREDENTIALS on its own, which keeps the key path out of your code.

Create the VPC

Next we need a VPC, because all the other resources depend on it. The following code creates a network with two subnetworks, one public and one private. The private subnet carries two secondary ranges: GKE assigns Pod IPs from pods and Service IPs from services, which makes the cluster VPC-native.

Because the nodes will be private (no external IPs), they need Cloud NAT to pull images from registries outside Google, such as Docker Hub. Save it as vpc.tf:

locals {
 vpc_name = "${local.env}-${local.project}-vpc"
}

resource "google_compute_network" "vpc" {
 name = local.vpc_name
 auto_create_subnetworks = false
}

resource "google_compute_subnetwork" "public" {
 name = "${local.vpc_name}-public-0"
 region = local.region
 network = google_compute_network.vpc.id
 ip_cidr_range = "10.1.0.0/24"
}

resource "google_compute_subnetwork" "private" {
 name = "${local.vpc_name}-private-0"
 region = local.region
 network = google_compute_network.vpc.id
 ip_cidr_range = "10.1.1.0/24"
 private_ip_google_access = true

 secondary_ip_range {
 range_name = "pods"
 ip_cidr_range = "10.20.0.0/16"
 }

 secondary_ip_range {
 range_name = "services"
 ip_cidr_range = "10.30.0.0/20"
 }
}

# Private nodes have no public IP, so they need Cloud NAT to pull images from outside Google
resource "google_compute_router" "router" {
 name = "${local.vpc_name}-router"
 region = local.region
 network = google_compute_network.vpc.id
}

resource "google_compute_router_nat" "nat" {
 name = "${local.vpc_name}-nat"
 router = google_compute_router.router.name
 region = local.region
 nat_ip_allocate_option = "AUTO_ONLY"
 source_subnetwork_ip_ranges_to_nat = "ALL_SUBNETWORKS_ALL_IP_RANGES"
}

Size the pods range generously. By default each node reserves a /24 from it (110 Pods per node), so a /16 gives you room for about 256 nodes.

Create the GKE cluster

Next we can create our cluster. We only want to use separately managed node pools, so we create the smallest possible default node pool and immediately delete it with remove_default_node_pool. The cluster goes in the private subnet.

release_channel lets GKE upgrade the control plane and nodes automatically on the REGULAR cadence.
private_cluster_config.enable_private_nodes gives nodes internal IPs only. The control plane keeps a public endpoint (enable_private_endpoint = false), restricted by master_authorized_networks_config to the networks you list. Replace 203.0.113.0/24 with your own public IP range.
workload_identity_config lets Pods use Google service accounts without node keys.
deletion_protection defaults to true in the google provider, which blocks terraform destroy. It is set to false here so you can tear the tutorial down; leave it on for anything you care about.

The node pool runs as its own service account with only the roles nodes need, instead of the Compute Engine default service account (which has Editor on the project).

Add this content to gke.tf:

locals {
 cluster_name = "${local.env}-${local.project}-gke"
 gke_num_nodes = 1
}

resource "google_service_account" "gke_nodes" {
 account_id = "${local.cluster_name}-nodes"
 display_name = "GKE nodes for ${local.cluster_name}"
}

resource "google_project_iam_member" "gke_nodes" {
 for_each = toset([
 "roles/container.defaultNodeServiceAccount",
 "roles/artifactregistry.reader",
 ])

 project = local.project_id
 role = each.value
 member = google_service_account.gke_nodes.member
}

resource "google_container_cluster" "gke" {
 name = local.cluster_name
 location = local.region

 remove_default_node_pool = true
 initial_node_count = 1
 deletion_protection = false

 network = google_compute_network.vpc.id
 subnetwork = google_compute_subnetwork.private.id

 release_channel {
 channel = "REGULAR"
 }

 ip_allocation_policy {
 cluster_secondary_range_name = "pods"
 services_secondary_range_name = "services"
 }

 private_cluster_config {
 enable_private_nodes = true
 enable_private_endpoint = false
 }

 master_authorized_networks_config {
 cidr_blocks {
 cidr_block = "203.0.113.0/24"
 display_name = "office"
 }
 }

 workload_identity_config {
 workload_pool = "${local.project_id}.svc.id.goog"
 }
}

# Separately Managed Node Pool
resource "google_container_node_pool" "gke_nodes" {
 name = "${local.cluster_name}-node-pool"
 location = local.region
 cluster = google_container_cluster.gke.name
 node_count = local.gke_num_nodes

 node_config {
 machine_type = "e2-medium"
 disk_size_gb = 50
 service_account = google_service_account.gke_nodes.email
 oauth_scopes = ["https://www.googleapis.com/auth/cloud-platform"]
 # spot = true

 labels = {
 env = local.env
 }

 tags = ["gke-node", "${local.project}-gke"]

 workload_metadata_config {
 mode = "GKE_METADATA"
 }
 }

 management {
 auto_repair = true
 auto_upgrade = true
 }
}

output "gke_endpoint" {
 value = google_container_cluster.gke.endpoint
 sensitive = true
}

output "gke_master_version" {
 value = google_container_cluster.gke.master_version
}

output "gke_node_pool_instance_groups" {
 value = google_container_node_pool.gke_nodes.instance_group_urls
}

A note on location: setting it to a region creates a regional cluster with a control plane and nodes in three zones, and node_count is per zone. gke_num_nodes = 1 therefore gives you three nodes. Set location to a zone like europe-west1-b for a cheaper single-zone cluster.

On google provider 8.x, terraform validate warns that remove_default_node_pool is deprecated. On GKE 1.36.4-gke.1469000 and later you can drop both remove_default_node_pool and initial_node_count and the cluster is created without a default pool. The pattern above still works on every GKE version, so it is the safe default until your release channel is past that version.

Step 3 – Autopilot instead of Standard

With Autopilot, Google manages the nodes for you: no node pools, no machine types, no node upgrades. You pay for the CPU, memory and storage your Pods request instead of for VMs. For most workloads that do not need custom node configuration (GPUs aside, privileged DaemonSets, specific kernels), it is the simpler choice.

To use Autopilot, replace the cluster, node pool, and node service account in gke.tf with:

resource "google_container_cluster" "autopilot" {
 name = "${local.env}-${local.project}-autopilot"
 location = local.region

 enable_autopilot = true
 deletion_protection = false

 network = google_compute_network.vpc.id
 subnetwork = google_compute_subnetwork.private.id

 ip_allocation_policy {
 cluster_secondary_range_name = "pods"
 services_secondary_range_name = "services"
 }

 private_cluster_config {
 enable_private_nodes = true
 enable_private_endpoint = false
 }

 master_authorized_networks_config {
 cidr_blocks {
 cidr_block = "203.0.113.0/24"
 display_name = "office"
 }
 }
}

Autopilot clusters are always regional and already have Workload Identity, auto-repair and auto-upgrade turned on, so those blocks are not needed.

	Standard	Autopilot
Nodes	You define node pools and machine types	Managed by Google
Billing	Per node VM, whether used or not	Per Pod resource request
Node SSH, privileged Pods, custom DaemonSets	Yes	Restricted
Best for	Special hardware, tight control, packing many small Pods	Most apps, small teams

Step 4 – Planning and applying changes

First initialize Terraform to download the required providers:

terraform init

Then format and validate to ensure that you have valid code without errors:

terraform fmt
terraform validate

For the Standard cluster, validate prints the deprecation warning described above, followed by:

Success! The configuration is valid, but there were some validation warnings
as shown above.

Then plan to confirm that the changes being introduced are what is expected, and save the plan:

terraform plan -out=tf.plan

Finally apply that saved plan to create the resources in GCP. A saved plan is applied without a confirmation prompt, since you already reviewed it:

terraform apply tf.plan

Creating a GKE cluster takes roughly 5 to 15 minutes. Delete tf.plan afterwards: saved plans can contain sensitive values and do not belong in Git.

Step 5 – Connecting to the cluster

Use gcloud to fetch cluster credentials into your kubeconfig. The region must match local.region:

gcloud container clusters get-credentials dev-citizix-gke \
 --region europe-west1 --project citizix-prj

Fetching cluster endpoint and auth data.
kubeconfig entry generated for dev-citizix-gke.

If this fails with gke-gcloud-auth-plugin ... not found, go back to step 1 and install the plugin.

Then check the cluster:

kubectl cluster-info
kubectl get nodes -o wide

You should see three Ready nodes (one per zone) with an internal IP and no external IP, which confirms the nodes are private. On Autopilot, kubectl get nodes may show no nodes until you deploy something.

Run a quick test workload to confirm the nodes can pull from Docker Hub through Cloud NAT:

kubectl create deployment hello --image=nginx:stable
kubectl rollout status deployment/hello
kubectl get pods -l app=hello -o wide
kubectl delete deployment hello

The Pod IP comes from the pods secondary range (10.20.0.0/16). If the Pod is stuck in ImagePullBackOff, Cloud NAT is missing or not covering the subnet.

If you manage several clusters, see How to merge multiple kubeconfig files into one.

Troubleshooting

Error 403: Kubernetes Engine API has not been used in project ...: enable container.googleapis.com (see Requirements) and wait a minute.
Cannot destroy cluster because deletion_protection is set to true: set deletion_protection = false, run terraform apply, then terraform destroy.
kubectl times out: your IP is not in master_authorized_networks_config. Add it and apply.
Quota errors on create: a regional cluster needs three times the CPUs and IPs of a zonal one. Check IN_USE_ADDRESSES and CPUS quotas in the region, or use a zonal location.

Cost

GKE is not free to leave running. You pay for:

A per-cluster management fee. The GKE free tier credit covers one zonal or Autopilot cluster per billing account; regional Standard clusters pay it.
The node VMs (Standard) or Pod resource requests (Autopilot), plus their persistent disks.
Cloud NAT gateway hours and processed data, and any load balancers your Services create.

Check the current numbers on the GKE pricing page. For a tutorial, a zonal cluster with one e2-medium node (or spot = true in node_config) keeps the bill small. When you are done:

terraform destroy

Conclusion

We were able to use Terraform to create a VPC and a private GKE cluster in GCP, with the option of Autopilot, and connect to it with kubectl. This allows us to create and destroy resources easily, at the same time bringing in the benefits of having infrastructure as code.

Terraform -target: Plan, Apply or Destroy Specific Resources

Mon, 15 Nov 2021 17:51:13 +0000

Terraform allows you to target specific resources when you plan, apply, or destroy your infrastructure. You can use Terraform’s -target option to target specific resources, modules, or collections of resources.

The option instructs Terraform to focus its planning, apply or destroy efforts only on the resource instances that match the given address and on any objects those instances depend on. That last part matters, and the examples below show it.

Targeting is useful when your configuration contains lots of resources and you do not want to apply all of it: you just want to run one specific resource, or a few, out of your Terraform configuration.

Everything here works the same with OpenTofu: replace terraform with tofu. The output in this guide was captured with OpenTofu 1.11, which prints OpenTofu where Terraform prints Terraform; the plan format is otherwise the same, though the wording of some warnings differs (Terraform’s targeting warning does not mention -exclude, which is OpenTofu-only).

Targeting is for exceptions, not routine use

Before the how-to, the warning. Every targeted plan prints this:

Warning: Resource targeting is in effect

You are creating a plan with either the -target option or the -exclude
option, which means that the result of this plan may not represent all of the
changes requested by the current configuration.

The -target and -exclude options are not for routine use, and are provided
only for exceptional situations such as recovering from errors or mistakes,
or when OpenTofu specifically suggests to use it as part of an error message.

Take it seriously. A targeted apply leaves your state out of step with your configuration until you run a full plan/apply again. Good reasons to target:

Recovering from a failed apply, or breaking a dependency cycle Terraform cannot resolve on its own.
Creating one resource first because another one needs its value at plan time (Terraform’s own error messages suggest -target for this).
Rebuilding or destroying a single broken resource without touching anything else.

If you find yourself targeting the same group of resources every day, that group probably wants to be its own root module with its own state.

The example configuration

To produce real output without a cloud account, this guide uses the local provider and the built-in terraform_data resource (Terraform 1.4+ and all OpenTofu versions). The addresses work exactly the same for aws_instance, google_container_cluster or anything else.

main.tf:

terraform {
 required_providers {
 local = {
 source = "hashicorp/local"
 version = "~> 2.5"
 }
 }
}

resource "local_file" "app_config" {
 filename = "${path.module}/out/app.conf"
 content = "version = ${terraform_data.app_version.output}\n"
}

resource "local_file" "db_config" {
 filename = "${path.module}/out/db.conf"
 content = "host = 10.0.1.10\n"
}

resource "terraform_data" "app_version" {
 input = "1.0.0"
}

resource "terraform_data" "worker" {
 count = 2
 input = "worker-${count.index}"
}

module "uat_files_bucket" {
 source = "./modules/bucket"
 name = "uat-files"
}

module "prod_files_bucket" {
 source = "./modules/bucket"
 name = "prod-files"
}

modules/bucket/main.tf, a stand-in for an S3 bucket module:

variable "name" {
 type = string
}

resource "local_file" "bucket" {
 filename = "${path.root}/out/${var.name}.bucket"
 content = "bucket = ${var.name}\n"
}

resource "terraform_data" "policy" {
 input = "${var.name}-policy"
}

Initialize it once. init does not take -target: it always installs every provider and module the configuration uses.

terraform init

Target a single resource

The address is <resource_type>.<name>:

terraform plan -target=local_file.app_config

app_config reads terraform_data.app_version, so Terraform pulls the dependency in too, even though you only named one resource:

 # local_file.app_config will be created
 # terraform_data.app_version will be created

Plan: 2 to add, 0 to change, 0 to destroy.

Dependencies are always included. Dependents (things that read the targeted resource) are not, which is why a targeted apply can leave them stale.

Target a resource inside a module

Prefix the address with module.<module_name>.:

terraform plan -target=module.uat_files_bucket.local_file.bucket

Where:

module.uat_files_bucket is the module block’s name in the calling configuration (not the module’s source directory).
local_file.bucket is the resource type and name inside the module.

To target everything in a module, stop at the module address:

terraform plan -target=module.uat_files_bucket

 # module.uat_files_bucket.local_file.bucket will be created
 # module.uat_files_bucket.terraform_data.policy will be created

Plan: 2 to add, 0 to change, 0 to destroy.

Target multiple resources

Repeat -target, mixing root resources and module resources freely:

terraform plan \
 -target=module.uat_files_bucket.local_file.bucket \
 -target=local_file.db_config

 # local_file.db_config will be created
 # module.uat_files_bucket.local_file.bucket will be created

Plan: 2 to add, 0 to change, 0 to destroy.

With many addresses, OpenTofu also accepts -target-file=targets.txt (one address per line).

Target a count or for_each instance

Instances have an index in square brackets. Quote the whole address so your shell does not treat the brackets as a glob:

terraform plan -target='terraform_data.worker[1]'

 # terraform_data.worker[1] will be created

Plan: 1 to add, 0 to change, 0 to destroy.

For for_each, the key is a string, so it needs double quotes inside the single quotes, for example -target='module.bucket["uat"]' or -target='aws_instance.web["api"]'.

Example with S3 bucket modules: plan and apply with a saved plan

Plan both bucket modules, save the plan, then apply exactly that plan. Bash brace expansion gives you two -target flags in one go:

terraform plan -target=module.{uat,prod}_files_bucket -out=tf.plan

which the shell expands to:

terraform plan -target=module.uat_files_bucket -target=module.prod_files_bucket -out=tf.plan

 # module.prod_files_bucket.local_file.bucket will be created
 # module.prod_files_bucket.terraform_data.policy will be created
 # module.uat_files_bucket.local_file.bucket will be created
 # module.uat_files_bucket.terraform_data.policy will be created

Plan: 4 to add, 0 to change, 0 to destroy.

Saved the plan to: tf.plan

The targets are recorded in the saved plan, so apply does not need them again:

terraform apply tf.plan

module.prod_files_bucket.terraform_data.policy: Creating...
module.uat_files_bucket.terraform_data.policy: Creating...
module.prod_files_bucket.local_file.bucket: Creating...
module.uat_files_bucket.local_file.bucket: Creating...
module.prod_files_bucket.terraform_data.policy: Creation complete after 0s [id=f1a2bda7-e770-cb9d-e2fa-c88420b123f2]
module.uat_files_bucket.terraform_data.policy: Creation complete after 0s [id=77a7fe95-e8a5-e46d-acfa-65227d716acd]
module.uat_files_bucket.local_file.bucket: Creation complete after 0s [id=c11f1ee7d6dfb956552ec0f6b8c1775d1de9e2d6]
module.prod_files_bucket.local_file.bucket: Creation complete after 0s [id=a8f1163f04d698706a95ed239ee60bedd16d5427]

Warning: Applied changes may be incomplete

The plan was created with the -target or the -exclude option in effect, so
some changes requested in the configuration may have been ignored and the
output values may not be fully updated. Run the following command to verify
that no other changes are pending:
 tofu plan

Apply complete! Resources: 4 added, 0 changed, 0 destroyed.

A plain terraform plan afterwards shows what the targeted apply skipped:

 # local_file.app_config will be created
 # local_file.db_config will be created
 # terraform_data.app_version will be created
 # terraform_data.worker[0] will be created
 # terraform_data.worker[1] will be created

Delete tf.plan once it is applied: a saved plan can contain sensitive values in cleartext, so keep it out of Git.

Replace (recreate) one resource with -replace

terraform taint is deprecated. To force one resource to be destroyed and recreated, use -replace:

terraform plan -replace='module.uat_files_bucket.terraform_data.policy'

On its own, -replace is not a target: the plan also includes every other pending change in the configuration. To replace one resource and do nothing else, combine it with -target:

terraform plan \
 -replace='module.uat_files_bucket.terraform_data.policy' \
 -target='module.uat_files_bucket.terraform_data.policy'

 # module.uat_files_bucket.terraform_data.policy will be replaced, as requested
-/+ resource "terraform_data" "policy" {
 ~ id = "77a7fe95-e8a5-e46d-acfa-65227d716acd" -> (known after apply)
 ~ output = "uat-files-policy" -> (known after apply)
 # (1 unchanged attribute hidden)
 }

Plan: 1 to add, 0 to change, 1 to destroy.

Destroy a specific resource or module

-target works with destroy (or plan -destroy to preview):

terraform plan -destroy -target=module.prod_files_bucket

 # module.prod_files_bucket.local_file.bucket will be destroyed
 # module.prod_files_bucket.terraform_data.policy will be destroyed

Plan: 0 to add, 0 to change, 2 to destroy.

Destroy runs the dependency graph the other way round: destroying a resource also destroys everything that depends on it. Targeting terraform_data.app_version takes app_config with it:

terraform plan -destroy -target=terraform_data.app_version

 # local_file.app_config will be destroyed
 # terraform_data.app_version will be destroyed

Plan: 0 to add, 0 to change, 2 to destroy.

Always read the destroy plan before running terraform destroy -target=....

If you want a resource gone from state but still running in the cloud (for example, to hand it to another configuration), that is terraform state rm, or a removed block in Terraform 1.7+ and OpenTofu 1.7+, not a targeted destroy.

OpenTofu: exclude instead of target

OpenTofu 1.9+ adds the inverse, -exclude: plan everything except the given address and whatever depends on it. It cannot be combined with -target.

tofu plan -exclude=module.prod_files_bucket

-exclude carries the same “exceptional use only” warning as -target.

Common mistakes

terraform init -target=... or terraform validate -target=...: neither command accepts -target. Only plan, apply, destroy and refresh do.
Using the module’s directory name: the address uses the name in the module "<name>" block, not source.
Unquoted brackets: -target=aws_instance.web[0] can fail in zsh with no matches found. Quote it.
Forgetting the full plan afterwards: always run an untargeted terraform plan once the emergency is over and make sure it is clean.

Conclusion

In this guide, we learnt how to use Terraform targets to make changes to selected resources: single resources, module resources, whole modules and count/for_each instances, plus -replace, targeted destroys, and OpenTofu’s -exclude. Use them to get out of trouble, then run a full plan to bring state and configuration back in line.

Create an AWS Lightsail Instance With Terraform or OpenTofu

Wed, 25 Aug 2021 10:56:18 +0000

In this guide we will explore how to launch resources in the AWS Lightsail platform using Terraform.

Lightsail is an easy-to-use virtual private server (VPS) provider that offers you everything needed to build an application or website for a cost-effective, monthly plan. It is provided by AWS as a simpler way to manage cloud: a fixed monthly price per instance, with the IP, firewall and DNS handled inside Lightsail instead of VPCs and security groups.

We will create an SSH key pair, an Ubuntu 24.04 instance, a static IP attached to it, and firewall rules, then print the SSH command as an output. The same code works with OpenTofu, the open-source fork of Terraform: replace terraform with tofu in every command. The configuration below was validated with OpenTofu 1.11 and AWS provider 6.x.

Prerequisites

Terraform 1.6+ or OpenTofu 1.6+ installed (see the OpenTofu install docs or Terraform install docs).
AWS credentials configured, for example with aws configure. The provider reads the same ~/.aws/credentials and AWS_PROFILE as the AWS CLI.
An SSH key pair. If you don’t have one:

ssh-keygen -t ed25519 -C "lightsail" -f ~/.ssh/id_ed25519

Pick a blueprint and bundle

A blueprint is the OS image and a bundle is the instance size and price. IDs change as AWS adds releases and new hardware generations, so list the current ones for your region with the AWS CLI:

aws lightsail get-blueprints --region eu-west-1 \
 --query 'blueprints[?type==`os` && isActive].[blueprintId,name,version]' --output table

aws lightsail get-bundles --region eu-west-1 \
 --query 'bundles[?isActive].[bundleId,price,cpuCount,ramSizeInGb,diskSizeInGb]' --output table

aws lightsail get-regions --include-availability-zones \
 --query 'regions[?name==`eu-west-1`].availabilityZones[].zoneName'

This guide uses ubuntu_24_04 and small_3_0. The original version of this post used centos_8, which reached end of life in 2021 and is no longer offered.

The configuration

Create a directory with the following files.

versions.tf pins the provider and sets the region:

terraform {
 required_version = ">= 1.6"

 required_providers {
 aws = {
 source = "hashicorp/aws"
 version = "~> 6.0"
 }
 }
}

provider "aws" {
 region = var.region
}

variables.tf. Set admin_cidr to your own public IP so SSH is not open to the whole internet:

variable "region" {
 type = string
 default = "eu-west-1"
}

variable "availability_zone" {
 type = string
 default = "eu-west-1a"
}

variable "ssh_public_key_path" {
 type = string
 default = "~/.ssh/id_ed25519.pub"
}

variable "admin_cidr" {
 description = "CIDR allowed to SSH in, e.g. your office IP with /32"
 type = string
 default = "203.0.113.10/32"
}

main.tf holds the key pair, instance, static IP and firewall:

resource "aws_lightsail_key_pair" "main" {
 name = "citizix-key"
 public_key = file(pathexpand(var.ssh_public_key_path))
}

resource "aws_lightsail_instance" "server" {
 name = "server0"
 availability_zone = var.availability_zone
 blueprint_id = "ubuntu_24_04"
 bundle_id = "small_3_0"
 key_pair_name = aws_lightsail_key_pair.main.name

 # IPv4 only: the port rules below restrict IPv4 CIDRs, and a dual-stack
 # instance would also need ipv6_cidrs set on every port.
 ip_address_type = "ipv4"

 user_data = <<-EOT
 #!/bin/bash
 apt-get update
 apt-get install -y nginx
 systemctl enable --now nginx
 EOT

 tags = {
 Env = "dev"
 ManagedBy = "opentofu"
 }
}

resource "aws_lightsail_static_ip" "server" {
 name = "server0-ip"
}

resource "aws_lightsail_static_ip_attachment" "server" {
 static_ip_name = aws_lightsail_static_ip.server.name
 instance_name = aws_lightsail_instance.server.name
}

resource "aws_lightsail_instance_public_ports" "server" {
 instance_name = aws_lightsail_instance.server.name

 port_info {
 protocol = "tcp"
 from_port = 22
 to_port = 22
 cidrs = [var.admin_cidr]
 }

 port_info {
 protocol = "tcp"
 from_port = 80
 to_port = 80
 }

 port_info {
 protocol = "tcp"
 from_port = 443
 to_port = 443
 }
}

A few notes:

aws_lightsail_key_pair with public_key imports your existing key. If you leave public_key out, Lightsail generates a key pair and the private key ends up in the state file, so avoid that.
user_data is the Lightsail launch script. It runs once on first boot; here it installs nginx. Changing it later replaces the instance.
Without a static IP, the public IP changes every time the instance is stopped and started. A static IP is free while it is attached to an instance.
aws_lightsail_instance_public_ports manages the whole firewall for the instance: any port not listed is closed, including the default SSH and HTTP rules Lightsail adds. Each port appears once; cidrs restricts the source and defaults to everywhere.

outputs.tf prints what you need to connect:

output "public_ip" {
 value = aws_lightsail_static_ip.server.ip_address
}

output "username" {
 value = aws_lightsail_instance.server.username
}

output "ssh_command" {
 value = "ssh ${aws_lightsail_instance.server.username}@${aws_lightsail_static_ip.server.ip_address}"
}

Initialize and validate

tofu init
tofu fmt
tofu validate

Initializing provider plugins...
- Finding hashicorp/aws versions matching "~> 6.0"...
- Installing hashicorp/aws v6.72.0...
- Installed hashicorp/aws v6.72.0 (signed, key ID 0C0AF313E5FD9F80)
...
OpenTofu has been successfully initialized!

Success! The configuration is valid.

init writes a .terraform.lock.hcl file that pins the exact provider version; commit it with your code.

Plan and apply

Save the plan to a file and apply exactly that file, so what gets created is what you reviewed:

tofu plan -out=lightsail.tfplan
tofu apply lightsail.tfplan

The plan should show five resources to add: the key pair, the instance, the static IP, the attachment and the public ports. The instance takes under a minute to come up. When apply finishes it prints the outputs:

tofu output ssh_command

Connect with the command it prints (ubuntu@<static-ip> for Ubuntu blueprints) and check nginx answers on http://<static-ip>/. Delete lightsail.tfplan afterwards; it is a snapshot that can contain sensitive values, so keep it out of Git.

Destroy

Lightsail bills by the hour until the instance is deleted, and a static IP that is no longer attached is billed too. Tear everything down with:

tofu destroy

Troubleshooting

InvalidInputException: ... blueprint ... not found or bundle not found – the ID is not offered in that region or was retired. Re-run the get-blueprints / get-bundles commands above.
availability_zone errors – the zone must be in the provider’s region (eu-west-1a for eu-west-1).
SSH times out – your IP is not in admin_cidr. Check your public IP and update the variable, then apply again.
Permission denied (publickey) – you are using the wrong user or key. Ubuntu blueprints use ubuntu, Debian uses admin, Amazon Linux uses ec2-user; pass -i ~/.ssh/id_ed25519 if the key is not the default.
No valid credential sources found – the provider cannot find AWS credentials. Run aws sts get-caller-identity to check, or export AWS_PROFILE.

Opentofu on Citizix, The Best blog for Linux, Opensource, Devops, Cloud and all things awesome tech.

Create AWS VPC Peering in the Same Account with Terraform

Prerequisites

Provider and variables

Look up the VPCs

Create the peering connection

Update the route tables on both sides

Allow traffic in the security groups

Outputs

Plan and apply

Verify the peering

Troubleshooting

Related content

Conclusion

Create a VPC Network and GKE Cluster in GCP with Terraform

Requirements

Step 1 – Install Terraform, gcloud and kubectl

Step 2 – Adding the project code

Providers and locals

Create the VPC

Create the GKE cluster

Step 3 – Autopilot instead of Standard

Step 4 – Planning and applying changes

Step 5 – Connecting to the cluster

Troubleshooting

Cost

Conclusion

Terraform -target: Plan, Apply or Destroy Specific Resources

Targeting is for exceptions, not routine use

The example configuration

Target a single resource

Target a resource inside a module

Target multiple resources

Target a count or for_each instance

Example with S3 bucket modules: plan and apply with a saved plan

Replace (recreate) one resource with -replace

Destroy a specific resource or module

OpenTofu: exclude instead of target

Common mistakes

Related content

Conclusion

Create an AWS Lightsail Instance With Terraform or OpenTofu

Related content

Prerequisites

Pick a blueprint and bundle

The configuration

Initialize and validate

Plan and apply

Destroy

Troubleshooting