Atlas now manages Databricks streaming tables with the new streaming_table block.
The new streaming_table block declares a Databricks streaming table: a table that a streaming query fills, refreshed by a pipeline Databricks provisions for it. Atlas creates it with CREATE STREAMING TABLE.
Streaming Tables
A streaming table takes the required schema and as attributes, and the optional comment, properties, tags, and depends_on attributes. Its blocks are:
- column blocks, whose columns can also take a default or an identity
- primary_key and foreign_key
- expect, a named expectation with expr and an optional on_violation of DROP_ROW or FAIL_UPDATE
- cluster or partition, but not both
- refresh, with one of every, cron with an optional time_zone, or on_update with an optional at_most
- row_filter
With permissions = true in the env's mode block, a permission block whose for references a streaming table grants privileges on it.
Example
The project file uses catalog scope on both URLs, and manages grants with permissions = true:
env "local" { url = "databricks://...?catalog=atlas_demo" dev = "databricks://...?catalog=atlas_demo_dev" migration { dir = "file://migrations" tx_mode = "none" } schema { src = "file://schema.hcl" mode { permissions = true } }}
The desired state declares the raw_orders table and a streaming table that reads it with STREAM(raw_orders), drops rows with a non-positive amount, and refreshes every hour. A permission block grants SELECT on the streaming table to a service principal:
schema "default" {} table "raw_orders" { schema = schema.default column "order_id" { null = false type = BIGINT } column "customer_email" { null = true type = STRING } column "region" { null = true type = STRING } column "amount" { null = true type = DECIMAL(10,2) } column "ordered_at" { null = true type = TIMESTAMP } primary_key { columns = [column.order_id] }} streaming_table "orders_stream" { schema = schema.default comment = "Orders ingested from raw_orders" properties = { "delta.appendOnly" = "true" } column "order_id" { null = false type = BIGINT comment = "The order key" } column "customer_email" { null = true type = STRING } column "region" { null = true type = STRING } column "amount" { null = true type = DECIMAL(10,2) } column "ordered_at" { null = true type = TIMESTAMP } primary_key "orders_stream_pk" { columns = [column.order_id] } cluster { columns = [column.region] } expect "positive_amount" { expr = "amount > 0" on_violation = DROP_ROW } expect "has_order_date" { expr = "ordered_at IS NOT NULL" } refresh { every = "1 HOUR" } as = "SELECT order_id, customer_email, region, amount, ordered_at FROM STREAM(raw_orders)"} permission { to = "d95538a9-6212-4a22-99c8-a16b08057d6b" for = streaming_table.orders_stream privileges = [SELECT]}
Run the diff:
atlas migrate diff --env localAtlas creates the table, then the streaming table that reads from it, and grants SELECT on it:
-- Create "raw_orders" tableCREATE TABLE IF NOT EXISTS `default`.`raw_orders` ( `order_id` BIGINT NOT NULL, `customer_email` STRING, `region` STRING, `amount` DECIMAL(10,2), `ordered_at` TIMESTAMP, CONSTRAINT `raw_orders_pk` PRIMARY KEY (`order_id`));-- Create streaming table "orders_stream"CREATE STREAMING TABLE IF NOT EXISTS `default`.`orders_stream` ( `order_id` BIGINT NOT NULL COMMENT 'The order key', `customer_email` STRING, `region` STRING, `amount` DECIMAL(10,2), `ordered_at` TIMESTAMP, CONSTRAINT `positive_amount` EXPECT (amount > 0) ON VIOLATION DROP ROW, CONSTRAINT `has_order_date` EXPECT (ordered_at IS NOT NULL), CONSTRAINT `orders_stream_pk` PRIMARY KEY (`order_id`))CLUSTER BY (`region`)COMMENT 'Orders ingested from raw_orders'TBLPROPERTIES ('delta.appendOnly' = 'true')SCHEDULE REFRESH EVERY 1 HOUR AS SELECT order_id, customer_email, region, amount, ordered_at FROM STREAM(raw_orders);-- Grant privileges on streaming table "orders_stream" to "d95538a9-6212-4a22-99c8-a16b08057d6b"GRANT SELECT ON TABLE `default`.`orders_stream` TO `d95538a9-6212-4a22-99c8-a16b08057d6b`;
Getting Started
The Databricks driver is part of Atlas Pro. Run atlas login to get started, and see the Databricks HCL reference for every attribute and block.