Deploying schema migrations to Kubernetes with Init Containers

In Kubernetes, Init Containers are specialized containers that run before app containers in a Pod. Init containers can contain utilities or setup scripts not present in an app image.

Init containers can be utilized to run schema migrations with Atlas before the application loads. Because init containers can use a container image different from the application, developers can use a purpose-built image that only contains Atlas and the migration scripts to run them. This way, less can be included in the application runtime environment, which reduces the attack surface from a security perspective.

Depending on an application's deployment strategy, multiple replicas of an init container may run concurrently. In the case of schema migrations, this can cause a dangerous race condition with unknown outcomes. To prevent this, in databases that support advisory locking, Atlas will acquire a lock on the migration operation before running migrations, making the operation mutually exclusive.

In this guide, we demonstrate how schema migrations can be integrated into a Kubernetes deployment using an init container.

Prerequisites to the guide:

1. A migrations docker image
2. A Kubernetes Deployment manifest defining your application.
3. A running Kubernetes cluster to work against.

Adding an init container

Suppose our deployment manifest looks similar to this:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
  labels:
    app: nginx
spec:
  replicas: 1
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx:1.14.2
        ports:
        - containerPort: 80

Now, let's say our migration container image which contains the Atlas binary and our migration scripts is available at ghcr.io/repo/migrations:v0.1.2. We would like to run migrate apply against our target database residing at mysql://root:s3cr37p455@dbhostname.io:3306/db.

We will use a Kubernetes Secret to store a config file containing the credentials to our database. Create the file:

atlas.hcl

env "k8s" {
  url = "mysql://root:s3cr37p455@dbhostname.io:3306/db"
}

Kubernetes accepts secrets encoded as base64 strings. Let's calculate the base64 string representing our project file:

cat atlas.hcl | base64

Copy the result:

ZW52ICJrOHMiIHsKICB1cmwgPSAibXlzcWw6Ly9yb290OnMzY3IzN3A0NTVAZGJob3N0bmFtZS5pbzozMzA2L2RiIgp9Cg==

Create the secret manifest:

apiVersion: v1
kind: Secret
metadata:
  name: atlas-project
type: Opaque
data:
  atlas.hcl: ZW52ICJrOHMiIHsKICB1cmwgPSAibXlzcWw6Ly9yb290OnMzY3IzN3A0NTVAZGJob3N0bmFtZS5pbzozMzA2L2RiIgp9Cg==

Apply the secret on the cluster:

kubectl apply -f secret.yaml

The secret is created:

secret/atlas-project created

Next, add a volume to mount the config file and an init container using it to the deployment manifest:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
  labels:
    app: nginx
spec:
  replicas: 1
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      volumes:
      - name: atlas-project
        secret:
          secretName: atlas-project
      initContainers:
      - name: migrate
        image: ghcr.io/repo/migrations:v0.1.2
        imagePullPolicy: Always
        args: ["migrate", "apply", "-c", "file:///etc/atlas/atlas.hcl", "--env", "k8s"]
        volumeMounts:
        - name: atlas-project
          mountPath: "/etc/atlas"
      containers:
      - name: nginx
        image: nginx:1.14.2
        ports:
        - containerPort: 80

Notice the new configuration blocks we added to our deployment manifest:

• We added our secret atlas-project as a volume to the the deployment's PodSpec.
• We add an initContainer named migrate that runs the ghcr.io/repo/migrations:v0.1.2 image.
• We mounted the atlas-project volume at /etc/atlas in our init container.
• We configured our init container to run with these flags: ["migrate", "apply", "-c", "file:///etc/atlas/atlas.hcl", "--env", "k8s"]

Wrapping up

That's it! After we apply our new deployment manifest, Kubernetes will first run the init container and only then run the application containers.