Back to changelog
New
•2 minute read

Databricks: Streaming Tables

Atlas now manages Databricks streaming tables with the new streaming_table block.

The new streaming_table block declares a Databricks streaming table: a table that a streaming query fills, refreshed by a pipeline Databricks provisions for it. Atlas creates it with CREATE STREAMING TABLE.

Streaming Tables

A streaming table takes the required schema and as attributes, and the optional comment, properties, tags, and depends_on attributes. Its blocks are:

With permissions = true in the env's mode block, a permission block whose for references a streaming table grants privileges on it.

Example

The project file uses catalog scope on both URLs, and manages grants with permissions = true:

atlas.hcl
env "local" {  url = "databricks://...?catalog=atlas_demo"  dev = "databricks://...?catalog=atlas_demo_dev"  migration {    dir     = "file://migrations"    tx_mode = "none"  }  schema {    src = "file://schema.hcl"    mode {      permissions = true    }  }}

The desired state declares the raw_orders table and a streaming table that reads it with STREAM(raw_orders), drops rows with a non-positive amount, and refreshes every hour. A permission block grants SELECT on the streaming table to a service principal:

schema.hcl
schema "default" {}
table "raw_orders" {  schema = schema.default  column "order_id" {    null = false    type = BIGINT  }  column "customer_email" {    null = true    type = STRING  }  column "region" {    null = true    type = STRING  }  column "amount" {    null = true    type = DECIMAL(10,2)  }  column "ordered_at" {    null = true    type = TIMESTAMP  }  primary_key {    columns = [column.order_id]  }}
streaming_table "orders_stream" {  schema  = schema.default  comment = "Orders ingested from raw_orders"  properties = {    "delta.appendOnly" = "true"  }  column "order_id" {    null    = false    type    = BIGINT    comment = "The order key"  }  column "customer_email" {    null = true    type = STRING  }  column "region" {    null = true    type = STRING  }  column "amount" {    null = true    type = DECIMAL(10,2)  }  column "ordered_at" {    null = true    type = TIMESTAMP  }  primary_key "orders_stream_pk" {    columns = [column.order_id]  }  cluster {    columns = [column.region]  }  expect "positive_amount" {    expr         = "amount > 0"    on_violation = DROP_ROW  }  expect "has_order_date" {    expr = "ordered_at IS NOT NULL"  }  refresh {    every = "1 HOUR"  }  as = "SELECT order_id, customer_email, region, amount, ordered_at FROM STREAM(raw_orders)"}
permission {  to         = "d95538a9-6212-4a22-99c8-a16b08057d6b"  for        = streaming_table.orders_stream  privileges = [SELECT]}

Run the diff:

atlas migrate diff --env local

Atlas creates the table, then the streaming table that reads from it, and grants SELECT on it:

migrations/20261118033043.sql
-- Create "raw_orders" tableCREATE TABLE IF NOT EXISTS `default`.`raw_orders` (  `order_id` BIGINT NOT NULL,  `customer_email` STRING,  `region` STRING,  `amount` DECIMAL(10,2),  `ordered_at` TIMESTAMP,  CONSTRAINT `raw_orders_pk` PRIMARY KEY (`order_id`));-- Create streaming table "orders_stream"CREATE STREAMING TABLE IF NOT EXISTS `default`.`orders_stream` (  `order_id` BIGINT NOT NULL COMMENT 'The order key',  `customer_email` STRING,  `region` STRING,  `amount` DECIMAL(10,2),  `ordered_at` TIMESTAMP,  CONSTRAINT `positive_amount` EXPECT (amount > 0) ON VIOLATION DROP ROW,  CONSTRAINT `has_order_date` EXPECT (ordered_at IS NOT NULL),  CONSTRAINT `orders_stream_pk` PRIMARY KEY (`order_id`))CLUSTER BY (`region`)COMMENT 'Orders ingested from raw_orders'TBLPROPERTIES ('delta.appendOnly' = 'true')SCHEDULE REFRESH EVERY 1 HOUR AS SELECT order_id, customer_email, region, amount, ordered_at FROM STREAM(raw_orders);-- Grant privileges on streaming table "orders_stream" to "d95538a9-6212-4a22-99c8-a16b08057d6b"GRANT SELECT ON TABLE `default`.`orders_stream` TO `d95538a9-6212-4a22-99c8-a16b08057d6b`;
Note: Atlas now inspects streaming tables, so atlas schema apply plans a DROP TABLE for a streaming table on the target that the desired state does not declare, including one a Lakeflow pipeline owns. Before the first schema apply after upgrading, declare the streaming tables Atlas should manage, exclude the rest with --exclude, and review the plan with --dry-run.

Getting Started

The Databricks driver is part of Atlas Pro. Run atlas login to get started, and see the Databricks HCL reference for every attribute and block.

featuredatabricksstreaming tablesexpectationsatlas pro