Redshift: Datashares as Code

Atlas now inspects, diffs, and migrates Amazon Redshift datashares, so the schemas and objects a cluster shares, and the namespaces and accounts allowed to consume them, live alongside your schema and are reconciled with the exact CREATE DATASHARE, ALTER DATASHARE, and GRANT USAGE statements.

A datashare is how a Redshift cluster exposes live, read-only data to another cluster, workgroup, or AWS account without copying it. Atlas now manages datashares as a first-class resource: the schemas and objects a share carries, and the consumers granted usage on it, are inspected, diffed, and migrated with the rest of your schema.

Enabling datashares

Datashares are excluded from inspection and schema management by default. Turn them on per environment with a schema.mode block:

env "redshift" {  url = getenv("REDSHIFT_URL")  dev = getenv("REDSHIFT_DEV_URL")
  schema {    src = "file://schema.rs.hcl"    mode "redshift" {      datashares = true   // Inspect and manage datashares    }  }}

A datashare belongs to the database rather than to a single schema, so Atlas inspects the datashares of the database regardless of which schemas you selected for inspection.

Defining a datashare

A datashare block holds one schema block per shared schema, and each one names the objects shared from it three ways: an explicit objects list, include_new for the whole schema including objects created later, or include_new_except for the whole schema less a few. The three are mutually exclusive. Tables, views, materialized views, and functions can all be shared:

datashare "sales_share" {  // Share a fixed set of objects from a schema.  schema {    for     = schema.sales    objects = [table.orders, view.summary]  }
  // Share everything in the schema, now and later.  schema {    for         = schema.logs    include_new = true  }
  // Share everything except the objects listed.  schema {    for                = schema.events    include_new_except = [table.audit]  }
  namespaces = ["00000000-0000-0000-0000-000000000000"]}

Redshift applies INCLUDENEW only to objects created after it is set, so Atlas adds the objects that already exist explicitly, and the share covers the whole schema from the first apply rather than from the next CREATE TABLE.

Sharing functions

Functions are the one shared object that is not named the way the others are. A datashare holds a function by name, so it is referenced once, without a signature, exactly like a table:

datashare "sales_share" {  schema {    for = schema.sales    // One reference, no signature: the share holds the name.    objects = [function.calc]  }  namespaces = ["00000000-0000-0000-0000-000000000000"]}

Redshift's DDL, on the other hand, identifies a function by its signature. So Atlas expands that single reference into one entry per overload, each with its argument types spelled out:

-- add functions "sales"."calc"(integer), "sales"."calc"(integer, character varying(256)) of datashare "sales_share"ALTER DATASHARE "sales_share" ADD FUNCTION "sales"."calc"(integer), "sales"."calc"(integer, character varying(256));

The expansion is resolved from the functions in scope: Atlas looks up every function carrying that name in the schema, whether it was declared in your HCL or inspected from the cluster. The same lookup is what keeps inspection honest in the other direction, because the Redshift catalog records only the name of a shared function and not which overload the share holds. Atlas re-reads the signatures from the schema, which is also why datashares are inspected after the contents of the schemas they draw from.

Granting consumers

A share nobody can consume is meaningless, so every datashare declares at least one consumer. Use namespaces for a consumer in the same AWS account, and accounts for one in another account, which then grants its own namespaces. public_accessible controls whether the share can be consumed by a publicly accessible cluster:

datashare "sales_share" {  public_accessible = false
  schema {    for     = schema.sales    objects = [table.orders]  }
  // A consumer in the same AWS account, by namespace.  namespaces = ["00000000-0000-0000-0000-000000000000"]
  // A consumer in another AWS account, which then grants its own namespaces.  accounts = ["000000000000"]}

Applying the change

Both workflows are supported, and the statements are the same either way:

# Declarative: reconcile the cluster with the desired state.atlas schema apply --env redshift
# Versioned: capture the change as a migration file.atlas migrate diff add_datashare --env redshift

Whether they land in a plan or in a migration file, Atlas orders the statements around what the cluster requires: a schema is added to the share before the objects shared from it, and removed after them. Consecutive object changes on the same schema are collapsed into a single ALTER DATASHARE carrying the whole list:

-- create datashare "sales_share"CREATE DATASHARE "sales_share" SET PUBLICACCESSIBLE FALSE;-- add schema "sales" to datashare "sales_share"ALTER DATASHARE "sales_share" ADD SCHEMA "sales";-- add tables "sales"."orders", "sales"."summary" of datashare "sales_share"ALTER DATASHARE "sales_share" ADD TABLE "sales"."orders", "sales"."summary";-- add schema "logs" to datashare "sales_share"ALTER DATASHARE "sales_share" ADD SCHEMA "logs";-- include new objects of schema "logs" in datashare "sales_share"ALTER DATASHARE "sales_share" SET INCLUDENEW = TRUE FOR SCHEMA "logs";-- grant usage on datashare "sales_share"GRANT USAGE ON DATASHARE "sales_share" TO NAMESPACE '00000000-0000-0000-0000-000000000000';

Every change carries its inverse, so plans stay reversible and atlas migrate down works on them. A revoked consumer becomes REVOKE USAGE ON DATASHARE, a removed object ALTER DATASHARE ... REMOVE TABLE, and a dropped share reverses into the full CREATE DATASHARE with its schemas, objects, and grants.

Going the other way works too: point Atlas at an existing cluster and atlas schema inspect writes the datashares it finds back out as HCL or SQL.

Getting Started

Redshift datashares are part of Atlas Pro:

$ atlas login

See the HCL reference for every attribute of the datashare block, and the Redshift security guide for how shares fit next to the roles, groups, and grants that Atlas already manages.