Skip to main content

Versioned Migrations with the Atlas Operator

The Atlas Kubernetes Operator supports versioned migrations. In versioned migrations, the database schema is defined by a series of SQL scripts ("migrations") that are applied in lexicographical order. The user can specify the version and migration directory to run, which can be located on the Atlas Cloud or stored as a ConfigMap in your Kubernetes cluster.

In this workflow, after installing the Atlas Kubernetes Operator, the user defines the desired state of the database as an AtlasMigration resource which connects between a target database and a migration directory. The migration directory may be configured as a remote directory in Atlas Cloud or as a ConfigMap in your Kubernetes cluster.

The operator then reconciles the desired state with the actual state of the database by applying any pending migrations on the target database.

This Document​

This document describes the declarative workflow supported by the Atlas Kubernetes Operator. It covers the following topics:

Providing Migrations​

The versioned workflow requires the user provide a valid Atlas Migration Directory. Migrations are typically automatically generated by Atlas using the migrate diff but can be provided manually as well.

This sections outlines the different ways in which users can provide a migration directory to the Atlas Operator.

From Schema Registry​

The recommended way of providing the schema to the AtlasMigration resource is by referencing a schema stored in the Atlas Schema Registry:

apiVersion: db.atlasgo.io/v1alpha1
kind: AtlasMigration
metadata:
  name: atlasmigration-sample
spec:
  url: "<db url>"
  cloud:
    tokenFrom:
      secretKeyRef:
        key: token
        name: atlas-credentials
  dir:
    remote:
      name: "atlas" # The name of the project on the Schema Registry
      tag: "commit-id" # When tag is not specified, the latest tag will be used

This is akin to specifying a tagged container image in a Kubernetes Deployment resource.

This approach has multiple benefits:

  • Directory sizes are not limited by the Kubernetes API server's size constraints.
  • Directories are produced by a CI/CD pipeline and treated as a build artifact, ensuring only validated schemas are applied.
  • Directory states can be correlated with Git commits as tags, ensuring application code and database schema are aligned.

The Schema Registry is a hosted service provided by Atlas that stores and serves schemas to the Atlas Kubernetes Operator. To facilitate access to the registry, the API token is stored in a Kubernetes secret and referenced in the AtlasMigration resource.

For a full example see our two part blog post on GitOps for Databases (part 1) and GitOps for Databases (part 2).

From ConfigMap​

The Atlas Operator supports defining the migration directory as ConfigMap with key-value pairs where the key is the file name and the value is the content of the migration file.

atlas-migration-configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: migration-dir
data:
  20230316085611.sql: |
    create table users (
      id int not null auto_increment,
      name varchar(255) not null,
      email varchar(255) unique not null,
      short_bio varchar(255) not null,
      primary key (id)
    );
  atlas.sum: |
    h1:sPURLhQRvLU79Dnlaw3aiU4KVkyUVEmW+ekenqu/V2o=
    20230316085611.sql h1:FKUFrD9E4ceSeBZ5owv2c05Ag8rokXGKXp53ZctbocE=

The AtlasMigration resource can then reference the relevant ConfigMap:

atlas-migration.yaml
apiVersion: db.atlasgo.io/v1alpha1
kind: AtlasMigration
metadata:
  name: atlasmigration-sample
spec:
  urlFrom:
    secretKeyRef:
      key: url
      name: db-credentials
  dir:
    configMapRef:
      name: "migration-dir" # ConfigMap name of atlas-migration-configmap.yaml

Inline​

The migration directory can also be provided inline in the AtlasMigration resource:

apiVersion: db.atlasgo.io/v1alpha1
kind: AtlasMigration
metadata:
  name: atlasmigration-sample
spec:
  urlFrom:
    secretKeyRef:
      key: url
      name: db-credentials
  dir:
dir:
  local:
    20230316085611.sql: |
      create table users (
        id int not null auto_increment,
        name varchar(255) not null,
        email varchar(255) unique not null,
        short_bio varchar(255) not null,
        primary key (id)
      );
    atlas.sum: |
      h1:sPURLhQRvLU79Dnlaw3aiU4KVkyUVEmW+ekenqu/V2o=
      20230316085611.sql h1:FKUFrD9E4ceSeBZ5owv2c05Ag8rokXGKXp53ZctbocE=

Providing credentials​

URLs​

In order to manage the schema of your database, you must provide the Atlas Kubernetes Operator with an Atlas URL to your database. The Atlas Kubernetes Operator will use this URL to connect to your database and apply changes to the schema. As this string usually contains sensitive information, we recommend storing it as a Kubernetes secret.

Both AtlasSchema and AtlasMigration resources support defining the connection URL as a string (using the url field) or as a reference to a Kubernetes secret (using the urlFrom field):

tip

If you are connecting to a database that is running in your Kubernetes cluster, note that you should you use a namespace-qualified DNS name to connect to it. Connections are made from the Atlas Kubernetes Operator's namespace, so you should use a DNS name that is resolvable from that namespace. For example, to connect to the mysql service in the app namespace use:

mysql://user:pass@mysql.app:3306/myapp

Schema and database bound URLs​

Notice that depending on your use-case, you may want to use either a schema-bound or a database-bound connection.

  • Schema-bound connections are recommended for most use-cases, as they allow you to manage the schema of a database schema (named database). This is akin to selecting a specific database using a USE statement.
  • Database-bound connections are recommended for use-cases where you want to manage multiple schemas in a single database at once.

Create a file named db-credentials.yaml with the following contents:

db-credentials.yaml
apiVersion: v1
kind: Secret
metadata:
name: mysql-credentials
type: Opaque
stringData:
  url: "mysql://user:pass@your.db.dns.default:3306/myapp"

This defines a schema-bound connection to a named database called myapp.

For examples of URLs for connecting to other supported databases see the URL docs

urlFrom​

The urlFrom field is used to load the URL of the target database from another resource such as a Kubernetes secret.

urlFrom:
    secretKeyRef:
      key: url
      name: mysql-credentials

url​

The url field is used to define the URL of the target database directly. This is not recommended, but may be useful for testing purposes.

url: "mysql://user:pass@localhost:3306/myapp"

credentials object​

Alternatively, you can provide the credentials for your database as a credentials object.

apiVersion: db.atlasgo.io/v1alpha1
kind: AtlasSchema
metadata:
  name: atlasschema-postgres
spec:
  credentials:
    scheme: postgres
    host: postgres.default
    user: root
    passwordFrom:
      secretKeyRef:
        key: password
        name: postgres-credentials
    database: postgres
    port: 5432
    parameters:
      sslmode: disable
  # ... rest of the resource

Note: hostFrom and userFrom are also supported to load the host and user from another resource.

API Reference​

Example AtlasMigration Resource​

apiVersion: db.atlasgo.io/v1alpha1
kind: AtlasMigration
metadata:
  name: atlasmigration-sample
spec:
  envName: prod # Logs reported to Atlas cloud with this env name.
  revisionsSchema: atlas_schema_revisions # For database bound connections, controls the schema for the history table.
  urlFrom:
    secretKeyRef:
      key: url
      name: db-credentials
  cloud:
    tokenFrom:
      secretKeyRef:
        key: token
        name: atlas-credentials
  dir:
    remote:
      name: "atlas" # Migration directory name in your atlas cloud project
      tag: "commit-id" # When tag is not specified, the latest tag will be used

baseline​

Sets the baseline migration for the target database. This is used when deploying Atlas to an existing database that was not previously managed by Atlas. See the docs for baseline migrations.

baseline: "20220811074144"

devURL​

Atlas relies on a Dev Database to normalize schemas and make various calculations. By default, the operator will spin a new database pod to use as the Dev Database. However, if you have special requirements (such as have it preloaded with certain extensions or configuration), you can use the devURL field to specify a URL to use as the Dev Database.

devURL: mysql://root:password@dev-db-svc.default:3306/dev

devURLFrom​

Alternatively, you can use the devURLFrom field to specify a key in a Secret that contains the Dev Database URL.

devURLFrom:
  secretKeyRef:
    key: dev-url
    name: myapp

envName​

Controls the environment name reported to Atlas Cloud for deployment runs. Available only when using a remote type directory. Defaults to kubernetes.

envName: prod

execOrder​

Controls how Atlas computes and executes pending migration files to the database. Supports three values:

  • linear (default)
  • linear-skip
  • non-linear

For a technical explanation about the different modes see Applying Migrations.

url / urlFrom​

See the Credentials section for more information on how to provide the URL of the target database.

protectedFlows.migrateDown​

Controls the operators behavior when encountering a down migration (e.g the target revision is lower than the current).

enabled​

protectedFlows:
  migrateDown:
    enabled: true

Enables down migrations for the AtlasMigration resource.

autoApprove​

protectedFlows:
  migrateDown:
    autoApprove: false

Skips approval flows for the down migrations. Option not allowed for remote directories.

revisionsSchema​

The revisionsSchema field is used to define the schema name for the revisions table. Details about the revisions table can be found in the Revisions Schema.

revisionsSchema: "atlas"