Skip to main content

Snowflake and dbt: Managing the Schema Layer

A dbt project on Snowflake describes what to build. Nothing in it describes the layer it builds on: the schemas' settings, the landing table ingestion writes to, the file format for its files, and the role dbt authenticates with. Those objects are usually set up once in a worksheet and never diffed again.

This page covers the Snowflake specifics of managing that layer with Atlas. The setup guide builds the equivalent project on ClickHouse and explains the exclude patterns in general, and Getting Started with Snowflake covers connection URLs, inspection, and the declarative and versioned workflows.

The examples on this page use a Snowflake Standard-edition account in AWS_US_EAST_1, Atlas v1.3.4-9352233-canary, dbt 1.12.5 and dbt-snowflake 1.12.0. Capabilities that need another edition or additional setup are marked as such instead of being shown.

Install Atlas​

The Snowflake driver ships in Atlas's extended build:

To download and install the custom release of the Atlas CLI, simply run the following in your terminal:

curl -sSf https://atlasgo.sh | ATLAS_FLAVOR="snowflake" sh

If a snowflake:// URL returns Error: sql/sqlclient: unknown driver "snowflake", the default binary is still first on your PATH.

Snowflake support, including roles and permissions, is available to Pro users and during a trial. Run atlas login to authenticate.

Authentication in this guide uses a Snowflake programmatic access token in the password position of the URL. Snowflake is phasing out single-factor password sign-in; a programmatic access token is one replacement, and key-pair authentication on a service user is another (see Snowflake's documentation on programmatic access tokens). The same token works as dbt's password, since dbt-snowflake does not yet offer a token authenticator.

Who Owns Which Layer​

In a medallion warehouse, raw data lands in Bronze, and dbt cleans it into Silver and shapes it into Gold. dbt owns those two layers: every run builds and replaces the tables and views its models define.

Bronze has no model behind it. The landing tables, the file format that describes their files, the schemas themselves, and the roles and grants around all three sit outside the dbt lifecycle.

LayerObjectsOwner
PlatformSchemas, roles, grantsAtlas
BronzeLanding tables, file formatsAtlas
Silver and goldModel tables and viewsdbt
Cross-cuttingMigration history, linting, drift detectionAtlas

Bronze, silver and gold sit on a platform layer of schemas, roles and grants that Atlas manages as code. Atlas also owns bronze; dbt builds silver from bronze and gold from silver.

Ingestion writes into Bronze and dbt builds Silver and Gold from it. The silver and gold schemas stay under Atlas management because Atlas creates them and holds their grants; their contents are dbt's.

Setting Up the Versioned Workflow​

Schemas live inside a database, making them a straightforward fit for the versioned workflow: a migration directory, a plan per change, and lint in review.

Start with the schemas themselves:

schema.layer.hcl
schema "PUBLIC" {
retention_time = 1
}

schema "BRONZE" {}
schema "SILVER" {}
schema "GOLD" {}

The Atlas configuration builds the URLs for the target and dev databases and scopes Atlas to what this project owns:

atlas.hcl
locals {
base = "snowflake://${getenv("SNOWFLAKE_USER")}:${getenv("SNOWFLAKE_TOKEN")}@${getenv("SNOWFLAKE_ACCOUNT")}"
args = "warehouse=${getenv("SNOWFLAKE_WAREHOUSE")}&role=${getenv("SNOWFLAKE_ROLE")}"
url = "${local.base}/${getenv("SNOWFLAKE_DATABASE")}?${local.args}"
dev = "${local.base}/${getenv("SNOWFLAKE_DEV_DATABASE")}?${local.args}"

// The boundary: dbt owns every object it materializes inside SILVER and
// GOLD, so Atlas ignores their contents. The schemas themselves stay managed.
exclude = [
"SILVER.*",
"GOLD.*",
]
}

// Database-resident objects: schemas and the bronze landing objects. Versioned,
// with a migration directory under review.
env "prod" {
url = local.url
dev = local.dev

// The schemas this project manages. Everything else in the account, including
// Atlas's own revisions schema, stays out of the diff.
schemas = ["PUBLIC", "BRONZE", "SILVER", "GOLD"]

schema {
src = "file://schema.layer.hcl"
mode "snowflake" {
roles = false
permissions = false
}
}

exclude = local.exclude

migration {
dir = "file://migrations"
}

lint {
destructive {
error = true
}
}
}

schemas is what makes this safe to run against a shared account. It names what the project manages, and everything else is out of scope. Drop it and a live diff proposes dropping Atlas's own revisions schema, which lives in the target database and appears in no schema file.

It does not replace exclude. The dbt models sit inside SILVER and GOLD, which are schemas this project manages, so without the two exclude patterns, diff proposes dropping both models.

Create an initial migration plan for applying the schemas to your target database:

atlas migrate diff init_schemas --env prod
migrations/20261008031850_init_schemas.sql
-- Create "BRONZE" schema
CREATE SCHEMA "BRONZE" DATA_RETENTION_TIME_IN_DAYS = 1;
-- Create "GOLD" schema
CREATE SCHEMA "GOLD" DATA_RETENTION_TIME_IN_DAYS = 1;
-- Create "SILVER" schema
CREATE SCHEMA "SILVER" DATA_RETENTION_TIME_IN_DAYS = 1;

Lint the migration to verify it's safe to run:

atlas migrate lint --env prod --latest 1
Analyzing changes until version 20261008031850 (1 migration in total):

-- analyzing version 20261008031850
-- no diagnostics found
-- ok (11.617875ms)

-------------------------
-- 56.88098318s
-- 1 version ok
-- 3 schema changes
Lint Timing

migrate lint took about a minute to run for three CREATE SCHEMA statements, and as long again for the two bronze statements below. Every plan round-trips through the dev database, and Snowflake DDL is not fast. Budget for it in CI.

Apply the migration to your database and check the status to ensure everything is up-to-date:

atlas migrate apply --env prod
Migrating to version 20261008031850 (1 migrations in total):

-- migrating version 20261008031850
-> CREATE SCHEMA "BRONZE" DATA_RETENTION_TIME_IN_DAYS = 1;
-> CREATE SCHEMA "GOLD" DATA_RETENTION_TIME_IN_DAYS = 1;
-> CREATE SCHEMA "SILVER" DATA_RETENTION_TIME_IN_DAYS = 1;
-- ok (5.833798055s)

-------------------------
-- 24.353187982s
-- 1 migration
-- 3 sql statements
atlas migrate status --env prod
Migration Status: OK
-- Current Version: 20261008031850
-- Next Version: Already at latest version
-- Executed Files: 1
-- Pending Files: 0

The Bronze Landing Zone​

Bronze is where ingestion writes and dbt only reads. The landing table and the file format that describes the incoming files are two ordinary objects in the same schema file:

schema.layer.hcl
file_format "CSV_FORMAT" {
schema = schema.BRONZE
type = CSV
options = {
FIELD_DELIMITER = ","
SKIP_HEADER = 1
}
}

table "EVENTS" {
schema = schema.BRONZE
column "EVENT_ID" {
null = false
type = NUMBER(38)
}
column "USER_ID" {
null = false
type = NUMBER(38)
}
column "EVENT_TYPE" {
null = false
type = VARCHAR(64)
}
column "OCCURRED_AT" {
null = false
type = TIMESTAMP_NTZ(9)
}
}

Plan the change:

atlas migrate diff bronze_landing --env prod
migrations/20261008032251_bronze_landing.sql
-- Create file format "CSV_FORMAT"
CREATE FILE FORMAT "BRONZE"."CSV_FORMAT" TYPE = CSV SKIP_HEADER = 1 ESCAPE_UNENCLOSED_FIELD = '\\' NULL_IF = ('\\N');
-- Create "EVENTS" table
CREATE TABLE "BRONZE"."EVENTS" (
"EVENT_ID" NUMBER NOT NULL,
"USER_ID" NUMBER NOT NULL,
"EVENT_TYPE" VARCHAR(64) NOT NULL,
"OCCURRED_AT" TIMESTAMP_NTZ NOT NULL
) DATA_RETENTION_TIME_IN_DAYS = 1;

Lint has something to say about it:

atlas migrate lint --env prod --latest 1
Analyzing changes from version 20261008031850 to 20261008032251 (1 migration in total):

-- analyzing version 20261008032251
-- multiple non-transactional statements detected:
-- L1: Found 2 statements that can't run in a transaction (lines 2, 4). A failure
during execution may leave the database in an intermediate state between
the previous version "20261008031850" and "20261008032251"
https://atlasgo.io/lint/analyzers#TX101
-- ok (1.821334ms)

-------------------------
-- 54.389255499s
-- 1 version with warnings
-- 2 schema changes
-- 1 diagnostic

Atlas lint reports TX101 for this file: its two statements cannot run in a single transaction, so a failure between them would leave the database between versions. The analyzer's documented remedies are to split such statements into separate migration files, or to add -- atlas:txmode none at the top of the file.

After applying, schema inspect returns the table and the file format; the four schema blocks that precede them in the output are omitted here. The planned statement and the inspected block carry the options Snowflake stores for this format: ESCAPE_UNENCLOSED_FIELD and NULL_IF at their defaults, and no FIELD_DELIMITER, since a comma is the default.

atlas schema inspect --env prod
table "EVENTS" {
schema = schema.BRONZE
retention_time = 1
column "EVENT_ID" {
null = false
type = NUMBER(38)
}
column "USER_ID" {
null = false
type = NUMBER(38)
}
column "EVENT_TYPE" {
null = false
type = VARCHAR(64)
}
column "OCCURRED_AT" {
null = false
type = TIMESTAMP_NTZ(9)
}
}
file_format "CSV_FORMAT" {
schema = schema.BRONZE
type = CSV
options = {
ESCAPE_UNENCLOSED_FIELD = "\\"
NULL_IF = ["\\N"]
SKIP_HEADER = 1
}
}

A Role for dbt, as Code​

dbt creates and drops objects in its target schemas on every run, so its grants there have to cover objects that do not exist yet. That rules out table-level grants in SILVER and GOLD and makes schema scope the workable form. The bronze table dbt reads already exists, so it gets a table grant:

schema.platform.hcl
role "DBT_RUNNER" {}
role "BI_READER" {}

permission {
to = role.DBT_RUNNER
for = DATABASE
privileges = [USAGE]
}

permission {
to = role.DBT_RUNNER
for = schema.BRONZE
privileges = [USAGE]
}

permission {
to = role.DBT_RUNNER
for = table.EVENTS
privileges = [SELECT]
}

permission {
to = role.DBT_RUNNER
for = schema.SILVER
privileges = [USAGE, CREATE_TABLE, "CREATE VIEW"]
}

permission {
to = role.DBT_RUNNER
for = schema.GOLD
privileges = [USAGE, CREATE_TABLE, "CREATE VIEW"]
}

permission {
to = role.BI_READER
for = schema.GOLD
privileges = [USAGE]
}

DBT_RUNNER gets USAGE on the database, SELECT on the bronze table it reads, and USAGE, CREATE TABLE and CREATE VIEW on the two schemas it builds into. BI_READER gets USAGE on GOLD. Its SELECT on the gold views comes from dbt's grants model config, shown in the next section.

Database grants are planned like any other grant, so the database USAGE is declared here: a database grant the file does not declare is planned as a REVOKE. Keeping a declared database grant needs Atlas v1.3.4 or later; v1.3.3 plans the REVOKE even when the grant is declared.

info

privileges takes the documented enum values and plain strings (reference). dbt materializes views, so "CREATE VIEW" goes in as a string alongside the enum values. List privileges by name: ALL is not accepted.

Why roles need their own environment​

Snowflake nests most of its objects. An account holds databases, a database holds schemas, and a schema holds the tables and file formats this guide has been creating. Roles, warehouses and users sit outside that nesting, directly on the account. That division sets the layout of this project.

This matters because of how planning is done. Atlas plans by materializing the desired state in a dev database, and in this setup, the dev database lives in the same account as the target (security guide). For anything that lives inside a database, that is exactly the isolation a plan needs: the dev copy of BRONZE is a different object from the real one. An account-level object has no such copy. There is one DBT_RUNNER in the account, and the dev database sits next to it.

So each half of the schema layer gets the workflow that fits it, and each gains something:

  • Schemas and bronze objects stay versioned. Changes arrive as migration files, which means a plan to review, atlas migrate lint in CI, and a history you can roll forward environment by environment.
  • Roles and grants go declarative. The file is the access model rather than a log of edits to it, and schema apply diffs it against the account at any time with no history to replay. An access change is then reviewable as a diff of who can do what.

Roles and grants therefore get their own file and their own environment, alongside the prod env from earlier:

atlas.hcl
// Account-level objects: roles and the grants they hold, kept declarative so
// they are diffed against the account rather than replayed from history.
env "platform" {
url = local.url
dev = local.dev

schema {
src = ["file://schema.layer.hcl", "file://schema.platform.hcl"]
mode "snowflake" {
roles = true
permissions = true
}
}

// Roles do not live in a schema, so a schema list cannot scope them. On a
// shared account, include keeps Atlas to this project's roles.
include = [
"BRONZE",
"SILVER",
"GOLD",
"PUBLIC",
"DBT_RUNNER[type=role]",
"BI_READER[type=role]",
]
exclude = local.exclude
}

The schema file comes along in src because the permission blocks reference the schemas by name.

This environment uses include where the prod env used schemas, and the difference is not stylistic. A schema list cannot name an account-level object. Point this environment at a schema list instead and schema inspect returns the roles and users of every other team on the account, because none of them live in a schema. include is the filter that reaches them.

If roles do end up in the migration directory instead, the setup works exactly once. The first migrate apply succeeds, and the next migrate diff fails: computing the directory's state replays every migration on the dev database, and the replayed CREATE ROLE collides with the role the first apply already created in the account.

atlas migrate diff check_idempotent --env prod
Error: sql/migrate: read migration directory state: sql/migrate: executing statement "CREATE ROLE \"BI_READER\";" from version "20261008043300": 002002 (42710): SQL compilation error:
Object 'BI_READER' already exists.

Applying the platform environment​

Preview the plan:

atlas schema apply --env platform --dry-run --skip-lint
Planning migration statements (12 in total):

-- create role "bi_reader":
-> CREATE ROLE "BI_READER";
-- create role "dbt_runner":
-> CREATE ROLE "DBT_RUNNER";
-- grant on database "analytics" to "dbt_runner":
-> GRANT USAGE ON DATABASE "ANALYTICS" TO ROLE "DBT_RUNNER";
-- grant on schema "bronze" to "dbt_runner":
-> GRANT USAGE ON SCHEMA "BRONZE" TO ROLE "DBT_RUNNER";
-- grant on table "events" to "dbt_runner":
-> GRANT SELECT ON TABLE "BRONZE"."EVENTS" TO ROLE "DBT_RUNNER";
-- grant on schema "gold" to "bi_reader":
-> GRANT USAGE ON SCHEMA "GOLD" TO ROLE "BI_READER";
-- grant on schema "gold" to "dbt_runner":
-> GRANT CREATE TABLE ON SCHEMA "GOLD" TO ROLE "DBT_RUNNER";
-- grant on schema "gold" to "dbt_runner":
-> GRANT CREATE VIEW ON SCHEMA "GOLD" TO ROLE "DBT_RUNNER";
-- grant on schema "gold" to "dbt_runner":
-> GRANT USAGE ON SCHEMA "GOLD" TO ROLE "DBT_RUNNER";
-- grant on schema "silver" to "dbt_runner":
-> GRANT CREATE TABLE ON SCHEMA "SILVER" TO ROLE "DBT_RUNNER";
-- grant on schema "silver" to "dbt_runner":
-> GRANT CREATE VIEW ON SCHEMA "SILVER" TO ROLE "DBT_RUNNER";
-- grant on schema "silver" to "dbt_runner":
-> GRANT USAGE ON SCHEMA "SILVER" TO ROLE "DBT_RUNNER";

With roles declared in HCL, Atlas's plan check would run the plan's role statements on the account through the dev connection, so every dry run and apply in this environment uses --skip-lint. Read the printed plan before you approve it.

atlas schema apply --env platform --skip-lint

Once approved, Atlas runs the same 12 statements. The dry run then reports Schema is synced, no changes to be made.

tip

A platform run that stops midway can leave objects in the dev database. Check that it holds only PUBLIC and INFORMATION_SCHEMA before the next run.

Three grants stay outside the platform environment. Run them once, wherever the account is provisioned:

GRANT USAGE ON WAREHOUSE TRANSFORM_WH TO ROLE DBT_RUNNER;
GRANT ROLE DBT_RUNNER TO ROLE <ATLAS_ROLE>;
GRANT ROLE DBT_RUNNER TO USER <DBT_USER>;

Warehouses are not a grant target (reference), and Snowflake requires USAGE on the warehouse before a role can run a query (see Snowflake's access control privileges reference). The second grant follows Snowflake's standard role hierarchy, in which the role that manages the platform, here the role Atlas connects with (<ATLAS_ROLE>), inherits the functional roles; it also lets Atlas see the objects dbt owns. The third gives dbt's user the role it connects as. With the three in place, the platform plan still reports Schema is synced: the environment does not see the warehouse grant or grants to roles outside include, and it leaves undeclared users alone.

warning

A programmatic access token can carry a role restriction, and then it can only assume that role. Pointing dbt at DBT_RUNNER with a token restricted to another role fails, even when DBT_RUNNER is granted to the user:

390186 (08001): Failed to connect to DB: <ACCOUNT>.snowflakecomputing.com:443. Role 'DBT_RUNNER' specified in the connect string is not granted to this user, or is not permitted for the credentials being used.

Issue the token for the role dbt should run as, or authenticate dbt with a key pair on a service user.

Changing a grant​

An access change is an edit to schema.platform.hcl. To give BI_READER USAGE on SILVER, add a block:

schema.platform.hcl
permission {
to = role.BI_READER
for = schema.SILVER
privileges = [USAGE]
}

The plan holds that one grant:

atlas schema apply --env platform --dry-run --skip-lint
Planning migration statements (1 in total):

-- grant on schema "silver" to "bi_reader":
-> GRANT USAGE ON SCHEMA "SILVER" TO ROLE "BI_READER";

atlas schema apply --env platform --skip-lint applies it, and the next dry run reports Schema is synced, no changes to be made. Removing the block plans the matching revoke:

Planning migration statements (1 in total):

-- revoke on schema "silver" from "bi_reader":
-> REVOKE USAGE ON SCHEMA "SILVER" FROM ROLE "BI_READER";

Atlas's Boundary With dbt​

dbt connects to the objects declared above and materializes into two of the schemas Atlas manages. Nothing in the dbt project describes their DDL, only where to find them:

profiles.yml
analytics_demo:
target: prod
outputs:
prod:
type: snowflake
account: "{{ env_var('SNOWFLAKE_ACCOUNT') }}"
user: "{{ env_var('SNOWFLAKE_USER') }}"
password: "{{ env_var('SNOWFLAKE_TOKEN') }}"
role: "{{ env_var('SNOWFLAKE_ROLE') }}"
warehouse: "{{ env_var('SNOWFLAKE_WAREHOUSE') }}"
database: "{{ env_var('SNOWFLAKE_DATABASE') }}"
schema: SILVER
threads: 4

By default dbt prefixes a custom schema with the target's schema, so schema='GOLD' would build SILVER_GOLD: a schema Atlas does not manage and the dbt role cannot create. The macro makes schema='GOLD' mean GOLD.

dbt connects as DBT_RUNNER (SNOWFLAKE_ROLE), with a programmatic access token restricted to that role:

dbt run
03:55:22  Found 2 models, 1 source, 546 macros
03:55:22 Concurrency: 4 threads (target='prod')
03:55:28 1 of 2 START sql table model SILVER.stg_events ................................. [RUN]
03:55:30 1 of 2 OK created sql table model SILVER.stg_events ............................ [SUCCESS 1 in 2.39s]
03:55:30 2 of 2 START sql view model GOLD.daily_active_users ............................ [RUN]
03:55:33 2 of 2 OK created sql view model GOLD.daily_active_users ....................... [SUCCESS 1 in 2.76s]
03:55:36 Completed successfully
03:55:36 Done. PASS=2 WARN=0 ERROR=0 SKIP=0 NO-OP=0 REUSED=0 TOTAL=2

The gold model's grants config gives BI_READER SELECT on the view each time dbt builds it. In the ANALYTICS database:

SHOW GRANTS ON VIEW GOLD.DAILY_ACTIVE_USERS;
privilege | granted_on | name                              | granted_to | grantee_name | grant_option | granted_by
----------+------------+-----------------------------------+------------+--------------+--------------+-----------
SELECT | VIEW | ANALYTICS.GOLD.DAILY_ACTIVE_USERS | ROLE | BI_READER | false | DBT_RUNNER
OWNERSHIP | VIEW | ANALYTICS.GOLD.DAILY_ACTIVE_USERS | ROLE | DBT_RUNNER | true | DBT_RUNNER

With those two objects in the account, Atlas has nothing to say about them:

atlas migrate diff check_after_dbt --env prod && \
atlas schema diff --env prod --from "env://url" --to "env://schema.src"
The migration directory is synced with the desired state, no changes to be made
Schemas are synced, no changes to be made.

Remove the two exclude lines and run the same diff against the same account, and it is a different story:

-- Drop "DAILY_ACTIVE_USERS" view
DROP VIEW "GOLD"."DAILY_ACTIVE_USERS";
-- Drop "STG_EVENTS" table
DROP TABLE "SILVER"."STG_EVENTS";

This is the failure mode the exclude list prevents. Without it, every dbt model looks like an object that drifted into the account, and Atlas plans to remove it.

The boundary holds when the models change, too. Adding a column to the gold model and rebuilding it leaves the diff clean:

dbt run --select daily_active_users
atlas schema diff --env prod --from "env://url" --to "env://schema.src"
Schemas are synced, no changes to be made.

Drift on the Layer Atlas Owns​

Schema drift occurs when the target database diverges from the source of truth. For example, someone with account access adds a column in a hurry:

ALTER TABLE ANALYTICS.BRONZE.EVENTS ADD COLUMN SOURCE_SYSTEM VARCHAR(32);

The check that stayed silent for the dbt model change reports this one:

atlas schema diff --env prod --from "env://url" --to "env://schema.src"
-- Modify "EVENTS" table to drop "SOURCE_SYSTEM" column
ALTER TABLE "BRONZE"."EVENTS" DROP COLUMN "SOURCE_SYSTEM";
caution

That statement is a description of the difference, not a remediation script. Dropping the column may be right, or the column may be something the code should adopt. Decide, then either revert the change or add it to the schema file and generate a migration.

Reverting the manual change returns the check to Schemas are synced, no changes to be made. For running this on a schedule, and for the pre-apply gate that blocks a deployment onto a drifted account, see the warehouse drift guide, whose exclude semantics are the same as above.

What Else Atlas Manages on Snowflake​

This example needed schemas, a table, a file format, roles and grants. The Snowflake HCL reference covers a good deal more:

The example behind this page ran on a Standard-edition account with a non-admin role, so parts of that list went unexercised here: warehouses, where inspect read one into HCL but planning a change collided with the account-level object on a same-account dev database; resource monitors, which need ACCOUNTADMIN; masking policies, row access policies and tags, which are Enterprise-edition features; stages, which planned inconsistently on this account and were left out; Snowpipe auto-ingest, which needs a cloud notification integration; external and Iceberg tables, which need external storage this account was not set up for; dynamic tables, which were not tried; and column-level lineage, which is an Atlas Cloud feature rather than part of a CLI run.

For account-level objects specifically, Atlas documents planning against a separate dev Snowflake account, so the planner has somewhere to materialize them that is not production (warehouses, external volumes). This example had one account to work with and did not try it.

Considerations​

  • This guide does not put dbt's models under Atlas. Materializations belong to dbt, and the exclude list keeps Atlas out of them. Managing a model's table with Atlas would mean moving it out of dbt first.
  • Atlas does not read dbt's manifest. It does not know about ref(), source(), or the model DAG, so it cannot tell you which models a bronze column change breaks. dbt build after the migration answers that.
  • Atlas does not run dbt. Atlas applies schema changes; your scheduler runs dbt.
  • Atlas does not replace dbt tests. dbt tests assert on data. Atlas schema tests assert on schema objects and migration behavior.
  • This guide does not manage Snowflake users. dbt connects as an existing user, which receives DBT_RUNNER when the account is provisioned. For user blocks and the rest of the access graph, see the Snowflake security guide.
  • Atlas does not hold your Snowflake credentials. Authentication material stays in your secret store, and three grants stay outside Atlas as well: the warehouse USAGE (warehouses are not a grant target, reference), the role hierarchy grant, and the dbt user's role grant. They belong to whatever provisions the account.

Next Steps​

Have questions? Feedback? Find our team on our Discord server.