Changing a Databricks struct column now plans one statement per field instead of failing, and more schemas apply on the first run and stay in sync.
Changing a struct column on Databricks now plans field-level statements instead of a type change Databricks rejects. Schemas that failed on their first apply, or planned the same change on every run, now apply once and stay in sync.
Struct columns change field by field
Add a field to a struct, and to a struct inside an array:
column "payload" { type = sql("STRUCT<a:INT, b:STRING, c:INT>")}column "items" { type = sql("ARRAY<STRUCT<x:INT, y:INT>>")}
Atlas used to plan a type change on the whole column, which Databricks rejects:
ALTER TABLE `retail_analytics`.`events` ALTER COLUMN `payload` TYPE STRUCT<a:INT, b:STRING, c:INT>;-- [CANNOT_UPDATE_FIELD.STRUCT_TYPE] Cannot update ... field `payload` type:-- Update a struct by updating its fields. SQLSTATE: 0A000
It now plans one statement per field:
ALTER TABLE `retail_analytics`.`events` ADD COLUMN `payload`.`c` INT;ALTER TABLE `retail_analytics`.`events` ADD COLUMN `items`.`element`.`y` INT;
The same applies to nested structs and to structs inside a MAP. A field that moves gets a FIRST or AFTER clause, and dropping a field turns on column mapping first, as dropping a column does.
Plans that apply on the first run
Atlas now orders statements the way Databricks requires them:
- A SQL function is created after the tables and views its body reads, instead of failing with TABLE_OR_VIEW_NOT_FOUND.
- A CHECK, CLUSTER BY, row filter, or mask that names a new column runs after the column is added.
- A primary key change drops the foreign keys that reference the key first, and adds them back after the new key exists.
- A type change on a column a CHECK names drops the constraint first and adds it back after:
ALTER TABLE `retail_analytics`.`orders` DROP CONSTRAINT `amount_positive`;ALTER TABLE `retail_analytics`.`orders` ALTER COLUMN `amount` TYPE DECIMAL(12,2);ALTER TABLE `retail_analytics`.`orders` ADD CONSTRAINT `amount_positive` CHECK (amount > 0);
Plans that stay in sync
Databricks stores INTEGER as INT, and LONG, SHORT, BYTE, REAL, and NUMERIC as BIGINT, SMALLINT, TINYINT, FLOAT, and DECIMAL. Atlas now reads each synonym as the stored name, so applying the same schema again plans nothing:
# Before: applying a column declared as type = INTEGER again, without a dev databaseError: changing the type of column "amount" requires Delta type widening # NowSchema is synced, no changes to be made
These no longer plan the same change on every run either:
- A SQL function with a RETURN (SELECT ...) body, which Databricks stores with an extra pair of parentheses.
- A materialized view whose PRIMARY KEY has no name.
Changing a view that declares a column list keeps its column comments: Atlas redefines it with CREATE OR REPLACE and applies its grants again.
More schemas inspect and diff
- VARIANT, INTERVAL, GEOMETRY, and GEOGRAPHY columns are written as the catalog reports them. Every diff of a schema that held one used to fail.
- A view owned by another user no longer aborts the inspection.
- ?schema=Sales in the URL finds the sales schema, as Unity Catalog stores schema names in lower case.
Lint flags SET NOT NULL
SET NOT NULL fails when existing rows hold NULL, and a plan that adds a NOT NULL column to an existing Delta table ends with one. atlas migrate lint now reports it as MF104 before the migration runs:
-- data dependent changes detected: -- L8: Modifying nullable column "c" to non-nullable might fail in case it contains NULL values https://atlasgo.io/lint/analyzers#MF104
Getting Started
The Databricks driver is part of Atlas Pro. Run atlas login to get started, and see the Databricks guide.