Databricks: Metric Views and Dev Database Improvements

Atlas now manages Databricks metric views. External tables work in every workflow, SQL schema files plan only the table properties they declare, and the dev block with a baseline is supported.

Atlas now inspects, diffs, and plans Databricks metric views. This release also fixes how Atlas uses the dev database on Databricks, where the dev catalog shares the Unity Catalog metastore with the target.

Metric views

A Databricks metric view defines dimensions and measures over a source table or view in YAML. In Atlas, it is a view block with metrics = true, and as holds the YAML definition instead of a query. A metric view over the orders table:

view "revenue_metrics" {  schema     = schema.retail_analytics  metrics    = true  comment    = "Revenue KPIs by region and day"  as         = <<-SQL  version: 0.1  source: retail_analytics.orders  dimensions:    - name: region      expr: region    - name: order_day      expr: DATE(placed_on)  measures:    - name: total_revenue      expr: SUM(total_amount)  SQL  depends_on = [table.orders]}

Atlas creates it after the objects its YAML reads from:

-- Create view named "revenue_metrics"CREATE OR REPLACE VIEW `retail_analytics`.`revenue_metrics`COMMENT 'Revenue KPIs by region and day'WITH METRICSLANGUAGE YAMLAS $$version: 0.1source: retail_analytics.ordersdimensions:  - name: region    expr: region  - name: order_day    expr: DATE(placed_on)measures:  - name: total_revenue    expr: SUM(total_amount)$$;

Changes to the definition and the comment are applied in place, so the view keeps its grants. Adding a measure plans an ALTER VIEW:

-- Modify the definition of view "revenue_metrics"ALTER VIEW `retail_analytics`.`revenue_metrics` AS $$version: 0.1source: retail_analytics.ordersdimensions:  - name: region    expr: region  - name: order_day    expr: DATE(placed_on)measures:  - name: total_revenue    expr: SUM(total_amount)  - name: order_count    expr: COUNT(*)$$;

How Atlas handles a metric view:

• The YAML is compared as a document. Databricks reorders keys, adds blank lines, and quotes some values when it stores a version 1.1 definition, and these rewrites are not reported as changes.
• A top-level comment key in the YAML is the view's comment, the same as the comment attribute. A changed comment is planned as COMMENT ON VIEW.
• The columns of a metric view come from its YAML, so Atlas does not inspect or compare them.
• The tables and views named in the YAML source and joins are created first.
• Databricks cannot turn a view into a metric view, or the reverse. Atlas plans that change as a drop and a create.

External tables in every workflow

Unity Catalog requires the path of an external table to be unique across the metastore. Once the target held an external table, every command that ran its CREATE TABLE ... LOCATION on the dev catalog failed: replaying the migration directory in migrate diff, migrate validate, migrate lint, and migrate test, and the plan analysis of schema apply:

$ atlas migrate diff v2 --env prodError: sql/migrate: read migration directory state: ... [INVALID_PARAMETER_VALUE.LOCATION_OVERLAP]Input path url 's3://bucket/events' overlaps with other external tables or volumes ...

The dev catalog now creates each external Delta table at a managed path, skips ALTER TABLE ... SET LOCATION, and inspection puts the declared LOCATION back. This applies to HCL schemas, SQL schema files, and migration files. Statements on the target run as written. Tables the dev catalog cannot create at a managed path, such as a USING PARQUET table or one without a column list, also run as written there.

SQL schemas keep only declared properties

Databricks writes delta.* defaults onto every table it creates. A SQL schema file identical to the live table used to plan those defaults, including an irreversible Delta protocol upgrade. A SQL schema now keeps only the properties it declares, in the TBLPROPERTIES of CREATE TABLE or in ALTER TABLE ... SET TBLPROPERTIES, as HCL schemas already did:

-- Before: atlas migrate diff --to file://schema.sql, against an identical tableALTER TABLE `retail_analytics`.`events` SET TBLPROPERTIES ('delta.minReaderVersion' = '3');ALTER TABLE `retail_analytics`.`events` SET TBLPROPERTIES ('delta.minWriterVersion' = '7');ALTER TABLE `retail_analytics`.`events` SET TBLPROPERTIES ('delta.enableRowTracking' = 'true');
-- AfterThe migration directory is synced with the desired state, no changes to be made

Managed tables have no location

On some metastores, such as Azure with managed storage, Unity Catalog reports a __unitystorage path for managed tables. Inspection read it into location, and a generated migration then turned the table into an external one, or failed on replay with NO_PARENT_EXTERNAL_LOCATION_FOR_PATH. Inspection now leaves the location out of managed tables:

# Before: atlas schema inspect, managed tabletable "events" {  location = "abfss://<metastore root>/.../tables/<table id>"  ...}
# Aftertable "events" {  ...}

The dev block and baseline

A dev "databricks" block used to fail with missing dev block handler for "databricks". It now loads, so a baseline can create the tables of schemas Atlas does not manage, for views to read on the dev catalog, or fixture rows for tests:

dev "databricks" "dev" {  url      = var.dev_url  baseline = file("baseline.sql")}
env "prod" {  url = var.url  dev = dev.databricks.dev.url}

With a baseline, the values Databricks assigns on its own, such as delta.* defaults, the __unitystorage path, and the catalog's default collation, stay out of the plan.

Also fixed

• A composite_schema with named schema blocks works with a dev URL that is not bound to a schema.
• VARCHAR(n) and CHAR(n) columns are inspected and accepted in HCL, and a length change is a type change. Inspecting a schema that holds one used to fail.

Getting Started

The Databricks driver is part of Atlas Pro. Run atlas login to get started.

See the view HCL reference and the Databricks guide.