Snowflake: File Format Support

Atlas now supports Snowflake file formats as a first-class resource, with references from stages and external tables.

Atlas now supports file formats as a first-class Snowflake resource. File formats can be defined, diffed, and migrated alongside other schema objects like tables, stages, and external tables.

Defining File Formats

The file_format block defines a named Snowflake file format with type-specific options. Supported types: CSV, JSON, PARQUET, AVRO, ORC, and XML.

file_format "csv_format" {
  schema  = schema.public
  type    = CSV
  comment = "Basic CSV format"
  options = {
    FIELD_DELIMITER = "|"
    NULL_IF         = ["\N"]
    SKIP_HEADER     = 1
    TRIM_SPACE      = true
  }
}

Stage References

Stages can reference a named file format instead of inlining format options:

file_format "csv_format" {
  schema = schema.public
  type   = CSV
}

stage "data_stage" {
  schema      = schema.public
  url         = "s3://bucket/data/"
  file_format = file_format.csv_format
}

External Table References

External tables can also reference named file formats:

file_format "csv_format" {
  schema = schema.public
  type   = CSV
}

stage "data_stage" {
  schema = schema.public
  url    = "s3://bucket/data/"
}

external_table "user_data" {
  schema      = schema.public
  file_format = file_format.csv_format
  location {
    stage = stage.data_stage
    path  = "users/"
  }
  column "id" {
    type = NUMBER
  }
  column "name" {
    type = VARCHAR
  }
}