ClickHouse: Named Collections

Atlas now manages ClickHouse named collections declaratively. Define them in HCL or SQL, and Atlas plans the right CREATE / ALTER / DROP NAMED COLLECTION statements. Versioned migrations omit values to keep secrets out of version control; the declarative workflow with sensitive = ALLOW also handles value rotations.

Named collections in ClickHouse centralize key-value pairs (typically credentials and configuration for integrations such as S3, PostgreSQL, Kafka, or other ClickHouse instances) so they can be referenced by name from tables, dictionaries, and table functions instead of being inlined repeatedly. Atlas now manages them as a first-class resource: define the desired state in HCL or SQL, and Atlas plans the right CREATE, ALTER, and DROP NAMED COLLECTION statements.

Defining a Named Collection

The HCL named_collection block accepts one or more item entries, each with a key and a value:

schema "default" {
  engine = Atomic
}

named_collection "s3_data" {
  item "access_key_id" {
    value = getenv("AWS_ACCESS_KEY_ID")
  }
  item "secret_access_key" {
    value = getenv("AWS_SECRET_ACCESS_KEY")
  }
}

The same resource in SQL:

CREATE NAMED COLLECTION s3_data AS
  access_key_id = 'AKIA...',
  secret_access_key = 'wJalr...';

Versioned Migrations: Secrets Stay Out of Version Control

In the versioned workflow, atlas migrate diff generates statements for adding the collection and for adding or removing keys, but intentionally omits the values to keep secrets out of version control:

-- Create named collection "s3_data"
CREATE NAMED COLLECTION `s3_data` AS `access_key_id` = '', `secret_access_key` = '';
-- Add "region" key
ALTER NAMED COLLECTION `s3_data` SET `region` = '';
-- Remove "region" key
ALTER NAMED COLLECTION `s3_data` DELETE `region`;

Values are populated at apply time (manually or through a secrets manager). Value-only changes do not produce a versioned migration, by design: rotating credentials should not require a code change.

Declarative Workflow: Managing Values

For full lifecycle management including value rotation, use atlas schema apply with sensitive = ALLOW on the schema mode:

env "local" {
  url = getenv("DATABASE_URL")
  dev = docker.clickhouse.dev.url
  schema {
    src = "file://schema.sql"
    mode {
      sensitive = ALLOW
    }
  }
}

Atlas compares the actual values stored in ClickHouse against the desired state, emits ALTER NAMED COLLECTION ... SET for any drift, and masks sensitive values in the plan output. The real values are still used during execution:

-- modify named collection "s3_data":
-> ALTER NAMED COLLECTION `s3_data` SET (sensitive) = (sensitive), (sensitive) = (sensitive);

Getting Started

Named collections require ClickHouse 23.10 or later and named_collection_control enabled for the connecting user on the server side. The ClickHouse Named Collections guide walks through the full setup, including the dev-database volume configuration and the rotation example end-to-end.