Defining Target Groups

In Atlas, a target group is a collection of target databases whose schema is managed together. In a database-per-tenant architecture, each tenant's database is a target database, and all tenant databases are grouped into a target group. However, you can also group databases by other criteria, such as environment (dev, staging, prod), region, or any other criteria that makes sense for your application.

For example, you might group all databases in the same region into a target group to ensure that schema changes are applied consistently across all databases in that region, or to group free-tier databases separately from paid-tier databases.

Target groups can be defined statically or dynamically loaded from an API endpoint or a database query.

Target groups are defined in the project's atlas.hcl file and are later used by the Atlas CLI during the deployment process to determine which databases to deploy to.

Let's review some examples of how to define target groups in Atlas.

`env` blocks and `for_each` meta-arguments

Before we jump into various techniques to define target groups, let's first understand the for_each meta-argument for environment blocks in Atlas.

Environment blocks (env blocks) are used in Atlas project files (atlas.hcl) to group configuration settings for a specific environment. Normally, an env block is used to define the URL of a single target database, like so:

env "dev" {
  url = "postgres://root:pass@localhost:5432/dev"
}

However, using the for_each meta-argument, it is possible to define multiple instances of a specific environment block by iterating over a list of values. For example:

locals {
  target_db_urls = [
    "postgres://root:pass@host-1:5432",
    "postgres://root:pass@host-2:5432",
  ]
}

env "targets" {
    for_each = toset(local.target_db_urls)
    url = each.value
}

When the for_each meta-argument is used, the env block is instantiated for each value in the list, and the each object is used to access the current value. In our case, we will get two instances of the target block, one for each URL in the target_db_urls list.

Dynamically Computing URLs

A technique commonly used in atlas.hcl files is to dynamically compile URLs by combining values from various sources. For instance, the database instance URL might be provided as an input variable, with the database name added to it dynamically. Here's an example:

variable "db_instance_url" {
  type = string
}

locals {
  tenants = ["acme_corp", "widget_inc", "wayne_enterprises", "stark_industries"]
}

env "tenants" {
  for_each = toset(local.tenants)
  url = urlsetpath(var.db_instance_url, each.value)
}

Let's review the code snippet above:

We define a variable db_instance_url that will be used as the base URL for the database instances. This variable is provided by the user when running the Atlas CLI by providing the --var flag.
We define a local variable tenants that contains a list of tenant names.
We define an env block named tenants that iterates over the tenants list. For each tenant, we set the url attribute to the result of the urlsetpath function, which combines the db_instance_url with the tenant name.

The urlsetpath function

The urlsetpath function is a helper function provided by Atlas that allows you to set the "path" part of a URL. For example:

urlsetpath("postgres://root:pass@localhost:5432", "mydb")
#  ↳ Evaluates to "postgres://root:pass@localhost:5432/mydb"

urlsetpath("mysql://localhost:3306", "mydb")
#  ↳ Evaluates to "postgres://root:pass@localhost:5432/mydb"

Loading data from local JSON files

Suppose our list of tenants is stored in a local file named tenants.json:

tenants.json
{
  "tenants": [
    "acme_corp",
    "widget_inc",
    "wayne_enterprises",
    "stark_industries"
  ]
}

We can load this data into our atlas.hcl file using the file and jsondecode functions:

atlas.hcl
locals {
  f = file("tenants.json")
  decoded = jsondecode(local.f)
  tenants = local.decoded.tenants
}

env "tenants" {
  for_each = toset(local.tenants)
  url = urlsetpath("postgres://root:pass@localhost:5432", each.value)
}

Next, we define an environment block for this target group that consumes the target_tenants local variable into the for_each argument:

atlas.hcl
env "tenants" {
  for_each = toset(local.target_tenants)
  url = urlsetpath("postgres://root:pass@localhost:5432", each.value)
}

Let's review the code snippet above:

We define a local variable f that reads the contents of the tenants.json file.
Next, we use the jsondecode function to parse the JSON content into a structured object.
We extract the tenants array from the decoded JSON object and store it in the tenants local variable.
Finally, we define an env block named tenants that iterates over the tenants list. For each tenant, we set the url attribute to the result of the urlsetpath function, which combines the base URL with the tenant name.

Loading Data from an API Endpoint

In some cases, you may want to load target groups dynamically from an API endpoint. For example, you might have a service tenant-svc that provides a list of tenant databases based on some criteria. Let's suppose this service's endpoints recieve the target group ID in the path, such as https://tenant-svc/api/target-group/{id} and return a simple JSON payload:

{
  "databases": [
    "acme_corp",
    "widget_inc",
    "wayne_enterprises",
    "stark_industries"
  ]
}

You can use the runtimevar data source with the http scheme to fetch this data and use it to define target groups.

Here's an example of how you might load tenant databases from an API endpoint:

var "group_id" {
    type = string
}

data "runtimevar" "tenants" {
  url = "http://tenant-svc/api/target-group/${var.group_id}"
}

locals {
  decoded = jsondecode(data.runtimevar.tenants)
  tenants = local.decoded.databases
}

env "tenants" {
  for_each = toset(local.tenants)
  url = urlsetpath("postgres://root:pass@localhost:5432", each.value)
}

Let's unpack this example:

We define a variable group_id that will be used to fetch the tenant databases from the API endpoint.
We use the runtimevar data source with the http scheme to fetch the tenant databases from the API endpoint.
We parse the JSON response using the jsondecode function and extract the databases array.
We define an env block named tenants that iterates over the tenants list. For each tenant, we set the url attribute to the result of the urlsetpath function, which combines the base URL with the tenant name.

By using the runtimevar data source with the http scheme, you can dynamically load target groups from an API endpoint and use them to define target groups in your Atlas project.

Loading data from a Database Query

In some cases, you may want to load target groups dynamically from a database query. For example, you might have a database schema for each tenant in some instance, and would like to retrieve the list from the database's native information_schema tables.

You can utilize the sql data source to fetch this data and use it to define target groups.

var "url" {
  type = string
}

locals {
    pattern = "tenant_%"
}

data "sql" "tenants" {
  url = var.url
  query = <<EOS
SELECT `schema_name`
  FROM `information_schema`.`schemata`
  WHERE `schema_name` LIKE ?
EOS
  args = [local.pattern]
}

env "prod" {
  for_each = toset(data.sql.tenants.values)
  url      = urlsetpath(var.url, each.value)
}

Let's break down this example:

We define a variable url that will be used to connect to the database.
We define a local variable pattern that contains a pattern to match the tenant schemas. In this case, we're looking for schemas that start with tenant_.
We use the sql data source to execute a query against the database. The query selects the schema_name from the information_schema.schemata table where the schema_name matches the pattern.
We define an env block named prod that iterates over the results of the query. For each schema name, we set the url attribute to the result of the urlsetpath function, which combines the base URL with the schema name.

Incorporating Sensitive Data

Essentially, defining target groups in Atlas is about dynamically compiling a list of URLs that represent the target databases. Database URLs often contain sensitive information, such as passwords, that should not be hardcoded in the atlas.hcl file, which is typically checked into version control.

To address this issue, Atlas provides mechanisms for loading credentials from external sources, such as environment variables or secret management systems. This allows you to keep your database credentials secure while still being able to define target groups dynamically. Learn more about working with secrets.

For the purpose of this example, suppose our database password is stored in an AWS Secrets Manager, created using the AWS CLI as follows:

aws secretsmanager create-secret \
  --name db-pass-demo \
  --secret-string "p455w0rd"

The AWS CLI returns:

{
    "ARN": "arn:aws:secretsmanager:us-east-1:1111111111:secret:db-pass-demo-aBiM5k",
    "Name": "db-pass-demo",
    "VersionId": "b702431d-174f-4a8f-aee5-b80e687b8bf1"
}

To retrieve the secret value we will use the runtimevar data source in the atlas.hcl file:

var "db_instance_url" {
  type = string
}

var "db_user" {
  type = string
}

data "runtimevar" "pass" {
  url = "awssecretsmanager://db-pass-demo?region=us-east-1"
}

locals {
  db_with_pass = urluserinfo(var.db_instance_url, var.db_user, data.runtimevar.pass.value)
  tenants = ["acme_corp", "widget_inc", "wayne_enterprises", "stark_industries"]
}

env "tenants" {
  for_each = toset(local.tenants)
  url = urlsetpath(local.db_with_pass, each.value)
}

Let's review what's going on here:

We define two variables, db_instance_url and db_user, which are used to construct the database URL.
We use the runtimevar data source to fetch the password from AWS Secrets Manager.
We define a local variable db_with_pass that combines the database URL, the username, and the password.
We define an env block named tenants that iterates over the tenants list. For each tenant, we set the url attribute to the result of the urlsetpath function, which combines the db_with_pass with the tenant name.

The urluserinfo function

The urluserinfo function is a helper function provided by Atlas that allows you to set the "userinfo" part of a URL. For example:

urluserinfo("postgres://localhost:5432", "root", "p455w0rd")
#  ↳ Evaluates to "postgres://root:p455w0rd@localhost:5432"

Next Steps

In this guide, we've explored various techniques for defining target groups in Atlas. By using env blocks and for_each meta-arguments, you can dynamically compile a list of target databases based on various criteria, such as tenant names, regions, or other factors.

In the next section, we will show how to use these target groups in the deployment process to ensure that schema changes are applied consistently across all databases in the group.

env blocks and for_each meta-arguments​

Dynamically Computing URLs​

Loading data from local JSON files​

Loading Data from an API Endpoint​

Loading data from a Database Query​

Incorporating Sensitive Data​

Next Steps​