How to use Terraform to manage Confluent Cloud Clusters, Topics and permissions

In this guide, we will learn how to use terraform to launch Confluent Cloud resources such as environments, clusters, topics and ACLs.

Confluent Cloud is a fully managed, cloud-native service Kafka service provider for connecting and processing all of your data, everywhere it’s needed.

Checkout:

Prerequisites

Ensure that you have met the following conditions before proceeding:

Have latest version of terraform installed
Have a GCP Account and secret manager enabled for managing secrets

Setting up Provider settings

Save the following as part of the provider

terraform {
  required_providers {
    confluent = {
      source  = "confluentinc/confluent"
      version = "1.0.0"
    }
  }
}

provider "confluent" {}

In the above settings, we are defining providers for confluent. You can optionally provide api keys like this:

provider "confluent" {
  cloud_api_key    = local.confluent_cloud_api_key  
  cloud_api_secret = local.confluent_cloud_api_secret 
}

But that will be added to the version management system. Leaving out the keys will use the ones defined in environment variable. So export them like this:

export CONFLUENT_CLOUD_API_KEY=<api_key>
export CONFLUENT_CLOUD_API_SECRET=<api_secret>

Finally, let us define some variables for re-use:

locals {
  gcp_region  = "europe-west6"
  env         = "dev"
  name        = "test-cluster"
}

Define the environment

An environment contains Kafka clusters and deployed components such as Connect, ksqlDB, and Schema Registry. You can define multiple environments for an Organizations, and there is no charge for creating or using additional environments. Different departments or teams can use separate environments to avoid interfering with each other.

Use this to create an environment

resource "confluent_environment" "env" {
  display_name = local.env
}

Create a cluster

A Cluster is the Kafka instance system that consists of several Brokers, Topics, and Partitions for both. You can have multiple clusters in an environment:

Create the cluster with this code:

resource "confluent_kafka_cluster" "cluster" {
  display_name = "${local.env}-${local.name}"
  availability = "SINGLE_ZONE"
  cloud        = "GCP"
  region       = local.gcp_region
  basic {}

  environment {
    id = confluent_environment.env.id
  }
}

Create an Admin service account for cluster

We need to create a service account that can be used to run admin operations like create topics in the system. Once created, the service account needs to be assigned some permissions. We will assign CloudClusterAdmin to our service account and generate an api key and secret for it.

resource "confluent_service_account" "sa" {
  display_name = "admin-${local.env}"
  description  = "Admin Service Account for ${local.env}"
}

resource "confluent_role_binding" "admin-role-bind" {
  principal   = "User:${confluent_service_account.sa.id}"
  role_name   = "CloudClusterAdmin"
  crn_pattern = confluent_kafka_cluster.cluster.rbac_crn
}

resource "confluent_api_key" "admin-api-key" {
  display_name = "${local.env}-admin-api-key"
  description  = "Admin API key for ${local.env}"
  owner {
    id          = confluent_service_account.sa.id
    api_version = confluent_service_account.sa.api_version
    kind        = confluent_service_account.sa.kind
  }

  managed_resource {
    id          = confluent_kafka_cluster.cluster.id
    api_version = confluent_kafka_cluster.cluster.api_version
    kind        = confluent_kafka_cluster.cluster.kind

    environment {
      id = confluent_environment.env.id
    }
  }

  depends_on = [
    confluent_role_binding.admin-role-bind
  ]
}

Creating topics

Kafka topics are the categories used to organize messages. Each topic has a name that is unique across the entire Kafka cluster. Messages are sent to and read from specific topics. In other words, producers write data to topics, and consumers read data from topics.

It is possible to have many topics within a confluent cluster. Topics can have properties like name, partitions and configurations. To ease creation of many topics, we will define a yaml file with these properties that we can loop and create the respective topics with the properties.

Define the properties in a file called topics.yaml.

---
- Citizix.user.signups:
    partitions: 12
    config:
      retention.ms: 3600000
- Citizix.user.login:
    partitions: 12
    config:
      retention.ms: 3600000

We can then loop them and create topics with the properties defined in confluent cloud:

locals {
  topics_map      = merge(yamldecode(file("./topics.yaml"))...)
}

resource "confluent_kafka_topic" "services" {
  for_each = local.topics_map
  kafka_cluster {
    id = confluent_kafka_cluster.cluster.id
  }
  topic_name       = each.key
  partitions_count = each.value.partitions
  config           = contains(keys(each.value), "config") ? each.value.config : {}
  rest_endpoint    = confluent_kafka_cluster.cluster.rest_endpoint
  credentials {
    key    = confluent_api_key.admin-api-key.id
    secret = confluent_api_key.admin-api-key.secret
  }
}

Create a Service account for each topic

Kafka also supports refined permissions, i.e. you can create a service account and permissions for each service. We can define this in a yaml file defining each service with a list of read and write permissions.

Define a permissions.yaml that we can use for the defined services above:

---
- users:
    read_topics:
    - Citizix.payments.topup
    write_topics:
    - Citizix.user.signups
    - Citizix.user.login
- payments:
    read_topics:
    - Citizix.user.login
    write_topics:
    - Citizix.payments.topup

We can then define the terraform code to create the defined service account and permissions for each:

locals {
  permissions_map = merge(yamldecode(file("./permissions.yaml"))...)
}

resource "confluent_service_account" "service" {
  for_each     = local.permissions_map
  display_name = "${each.key}-${local.env}"
  description  = "Service Account for ${each.key} ${local.env}"
}

resource "confluent_api_key" "service-api-key" {
  for_each     = local.permissions_map
  display_name = "${each.key}-${local.env}-api-key"
  description  = "API key for ${each.key}-${local.env}"
  owner {
    id          = confluent_service_account.service[each.key].id
    api_version = confluent_service_account.service[each.key].api_version
    kind        = confluent_service_account.service[each.key].kind
  }

  managed_resource {
    id          = confluent_kafka_cluster.cluster.id
    api_version = confluent_kafka_cluster.cluster.api_version
    kind        = confluent_kafka_cluster.cluster.kind

    environment {
      id = confluent_environment.env.id
    }
  }

  depends_on = [
    confluent_service_account.service
  ]
}

If you have many services, the resulting service accounts and keys will be hard to manage. You can opt to use a secrets manager to manage them.

Here is an example of adding the generated service account details to google secret manager:

resource "google_secret_manager_secret" "service-kafka-api-key" {
  project   = local.gcp_project
  for_each  = local.permissions_map
  secret_id = "sm-app-${each.key}-kafka"

  replication {
    user_managed {
      replicas {
        location = local.gcp_region
      }
    }
  }
}

resource "google_secret_manager_secret_version" "service-kafka-api-key" {
  for_each = local.permissions_map
  secret   = google_secret_manager_secret.service-kafka-api-key[each.key].id

  secret_data = <<EOT
{
  "CC_KAFKA_ENDPOINT" : "${replace(confluent_kafka_cluster.cluster.bootstrap_endpoint, "SASL_SSL://", "")},
  "USERNAME" : "${confluent_api_key.service-api-key[each.key].output.key}",
  "PASSWORD" : "${confluent_api_key.service-api-key[each.key].output.secret}"
}
EOT
}

Running the code

In this guide, we learnt how to manage confluent cloud resources with terraform.