Skip to main content

Running Data Scripts on Kubernetes

Data Scripts run anywhere the Atlas CLI runs, including inside your cluster. The shape is always the same: the script is pushed to the Atlas Registry and pulled by name with an atlas:// URL, its credentials live in a Secret, and a pod running the arigaio/atlas image invokes script query, script exec, or script loop as its args. The pod holds no script source, so changing what the script does is a push, not a redeploy. Wrap that pod in a CronJob for recurring work, or in a Job for a one-off backfill or purge.

This guide builds the recurring case end to end with a monitor as its example. Every hour, it counts the orders stuck in pending and posts to a Slack channel when the result is higher than a specified threshold. The script contains three blocks: a query that reads the number, a break that ends the run when the number passes, and an http block that reports it when it does not. This way, a quiet hour sends nothing.

Atlas Pro Feature

Data Scripts are available to Atlas Pro users that purchased Atlas Pipelines. To use this feature, run:

atlas login

Prerequisites​

  • A Kubernetes cluster and kubectl, local (minikube, kind) or remote.
  • The Atlas CLI, logged in with atlas login, to push the script.
  • A database reachable from the cluster. The examples use PostgreSQL, and the script is engine-agnostic.
  • A Slack bot token with the chat:write scope, with the bot invited to the channel it posts to.

Step 1: Write the Monitor​

Save the script as monitor.script.hcl. The queue size is read once by a bound query, and both the break and the Slack message read it from there:

monitor.script.hcl
variable "slack_token" {
  type = string
}

variable "slack_channel" {
  type = string
}

variable "threshold" {
  type    = number
  default = 100
}

script "query" "pending_orders" {
  description = "Alerts Slack when the pending orders queue grows past the threshold."

  # Bound: prints nothing, feeds the two blocks below.
  query "queue" {
    sql = "SELECT count(*) AS pending FROM orders WHERE status = 'pending'"
    rows {
      pending = int
    }
  }

  # The healthy case: stop here, and say why.
  break "under_threshold" {
    expr    = query.queue.rows[0].pending <= var.threshold
    message = format("${query.queue.rows[0].pending} pending orders, at or under the threshold of %d", var.threshold)
  }

  # Reached only when the queue is over the threshold.
  http "slack" {
    url    = "https://slack.com/api/chat.postMessage"
    method = POST
    headers = {
      "Content-Type"  = "application/json"
      "Authorization" = "Bearer ${var.slack_token}"
    }
    body = jsonencode({
      channel = var.slack_channel
      text    = ":rotating_light: ${query.queue.rows[0].pending} orders are pending, over the threshold of ${var.threshold}."
    })
    expect_status = 200
    response      = object({ ok = bool, error = string })
    check {
      condition     = http.slack.ok == true
      error_message = "slack rejected the message: ${jsonencode(http.slack)}"
    }
  }

  output {
    message = "alerted on ${query.queue.rows[0].pending} pending orders"
  }
}

Three details carry the monitor:

  • The query declares rows, so it emits nothing and binds its result as query.queue.rows[0].pending. A printing query serializes to the output and is not in scope for the blocks after it.
  • The break uses expr, evaluated in-process over the count already read, so the healthy path costs one query and no second round trip. Its message is emitted in place of the script's output, so a quiet run still says what it saw.
  • Slack answers 200 and reports a refusal, such as an invalid token or a channel the bot is not in, in the body. expect_status catches the transport failure and the check over the decoded response catches that one, so a silent alert fails the run instead of passing.

Test it against the database before it goes anywhere near a cluster:

atlas script query \
  --url "$DB_URL" \
  --file "file://monitor.script.hcl" \
  --run '^pending_orders$' \
  --var slack_token="$SLACK_TOKEN" \
  --var slack_channel="C0123456789" \
  --var threshold=100 \
  --quiet

With the queue healthy, the run stops at the break and nothing is sent:

42 pending orders, at or under the threshold of 100

Lower --var threshold=0 to force the alerting path and confirm the message arrives in the channel:

alerted on 42 pending orders

Step 2: Push the Script​

Push the script to the Atlas Registry so the cluster can pull it by name:

atlas script push --file "file://monitor.script.hcl" db-monitors

The command prints a link to the script repository in Atlas Cloud. Every command that takes --file now accepts atlas://db-monitors in place of the local path:

atlas script query --url "$DB_URL" --file "atlas://db-monitors" --run '^pending_orders$' --quiet

Pushing again publishes a new version of the repository, and the CronJob picks it up on its next run. Tuning the threshold or widening the query is a push, with no change to any manifest.

Step 3: Create the Secret​

The job needs three credentials: an Atlas Cloud bot token, the Slack bot token, and the database URL. The bot token is what activates Atlas Pro inside the cluster and grants read access to the script repository. Create one in Atlas Cloud under ☰ > Settings > Bots, then put all three in one secret:

kubectl create secret generic atlas-monitor \
  --from-literal=atlas_token='aci_...' \
  --from-literal=slack_token='xoxb-...' \
  --from-literal=db_url='postgres://user:pass@postgres:5432/app?sslmode=disable'

The db_url is resolved from inside the cluster, so it names the database Service, not localhost. See URLs for the format each engine expects.

Step 4: Create the CronJob​

The arigaio/atlas image has the Atlas binary as its entrypoint, so the manifest passes the command as args:

cronjob.yaml
apiVersion: batch/v1
kind: CronJob
metadata:
  name: pending-orders-monitor
spec:
  # Every hour, on the hour.
  schedule: "0 * * * *"
  # A slow run never overlaps the next tick.
  concurrencyPolicy: Forbid
  successfulJobsHistoryLimit: 3
  failedJobsHistoryLimit: 3
  jobTemplate:
    spec:
      backoffLimit: 2
      template:
        spec:
          restartPolicy: Never
          containers:
            - name: atlas
              image: arigaio/atlas:latest
              args:
                - script
                - query
                - --url
                - $(DB_URL)
                - --file
                - atlas://db-monitors
                - --run
                - ^pending_orders$
                - --var
                - slack_token=$(SLACK_TOKEN)
                - --var
                - slack_channel=C0123456789
                - --var
                - threshold=100
              env:
                # Read by the Atlas CLI itself to authenticate with Atlas Cloud.
                - name: ATLAS_TOKEN
                  valueFrom:
                    secretKeyRef:
                      name: atlas-monitor
                      key: atlas_token
                - name: DB_URL
                  valueFrom:
                    secretKeyRef:
                      name: atlas-monitor
                      key: db_url
                - name: SLACK_TOKEN
                  valueFrom:
                    secretKeyRef:
                      name: atlas-monitor
                      key: slack_token

Apply it:

kubectl apply -f cronjob.yaml

Two things to note about the args:

  • Kubernetes expands $(VAR) in args from the container's own environment, which is how the secret values reach the flags without a shell. A $ that is not followed by (, such as the one anchoring ^pending_orders$, is left alone.
  • ATLAS_TOKEN is read by the CLI directly and needs no flag. The Slack token is a script variable, so it is passed with --var.

Rather than wait for the hour, trigger a run now:

kubectl create job --from=cronjob/pending-orders-monitor monitor-manual
kubectl logs job/monitor-manual

On a healthy queue the pod completes with the break message and no Slack call:

Executing script "pending_orders" (db-monitors:1):

  -- break "under_threshold"

  42 pending orders, at or under the threshold of 100

Over the threshold, the http step runs and the message lands in the channel. A rejected call, an unreachable database, or a token the registry does not accept exits non-zero, which Kubernetes surfaces as a failed job, so the monitor's own failures are visible to whatever already watches your jobs.

Pin the image

arigaio/atlas:latest is convenient for a first run. Pin a released tag for anything scheduled, so an hourly job does not change behavior underneath you.

Next Steps​

The manifest is the same for every Data Script. Only the verb, the --run pattern, and the variables change, so once a script runs in the cluster the choice of what it does is a question about the script, not the deployment:

  • Queries and Reports covers the kind used here, along with the rest of what a query script can do: printing results as CSV or a table, more break conditions, and richer http calls.
  • Transactional Mutations covers script exec for writes that must land as one unit, with condition, assert, and check guards and automatic rollback.
  • Batched Loops covers script loop for purges and backfills that work through the rows in batches. These usually belong in a one-off Job instead of a CronJob: the same pod spec under apiVersion: batch/v1, kind: Job, with backoffLimit: 0 so work that is not safe to rerun is not retried automatically.

Whichever kind you run, Masking Sensitive Output redacts columns before they leave Atlas, and Testing Data Scripts asserts the script's behavior in CI before it reaches a schedule.