Your Software Factory Stops at the Database
In August, I wrote that the AI-native SDLC playbook stops at the database. Code is cheap to regenerate. The data in your production database isn't. Software factories make this sharper. An agent takes a ticket, writes the code, runs the tests, and ships it, with no human in the loop. That works for code because code has gates. Tests say whether it's correct, and a bad deploy rolls back.
A migration has no such gate. It can pass every test and still lock your largest table for minutes, because what it costs depends on the data, not the code. Today, the gate is a person who reads the migration and knows which tables are big. A factory doesn't have that person.
In this post, I'll show how we close that gap in Atlas. The team writes down what database code may cost: migrations, queries, functions, procedures, and triggers. Atlas measures every agent's change against those limits twice, first on synthetic data in the agent's sandbox, then on a masked production snapshot before deploy.
Correct Is Not the Same as Safe
Migration linting catches known patterns, like an
index created without CONCURRENTLY, and enforces the team's policies, such
as no destructive changes. That's governance, and a factory still needs it. Tests check that the code
properly handles the data. Neither tells you how long a statement will hold a lock, because
that depends on how many rows the table has and what they look like.
The expensive cases are ordinary ones:
- Changing a column type can rewrite the whole table.
- Building an index sorts every row and spills to disk once the sort outgrows its memory.
- Adding
NOT NULLor aCHECKconstraint scans the table under a lock. - Recreating a materialized view runs its query again.
- A backfill touches every row its
WHEREclause matches. - A query, function, or trigger that filters on a column with no index reads every row, each time it runs.
On an empty dev database, these take milliseconds to complete, because an agent only sees the schema, not the 300 million rows behind it.
In our experience, the common practice is to let agents add tables and nullable columns, and send everything else to a person. But the same change can be safe on one table and cause an outage on another. An index on a 10,000-row table is nothing. On a 500 GB table, it's an incident. And even with a person in the loop, reading the code doesn't predict the impact.
The question isn't what kind of change it is. It's what the change costs on your data and your database, in the locks it holds and whether it chokes the server or runs it out of disk space.
Write the Limits Down
We recently added profile expectations to the Atlas testing
framework. An expectation is a bound on what a step of a test does to the database, and the test
fails when the step crosses it. The team writes them once, in a .test.hcl file:
expect "profile" "prod_safe" {
# Never rewrite a table.
metric "db.table.rewrites" {
eq = 0
}
# Nothing blocks reads or writes for more than a second.
metric "db.lock.time" {
le = "1s"
}
# Cap the WAL a single migration writes.
metric "db.log.written" {
le = "2GB"
}
}
A bound can cover rows read or changed, table rewrites and the disk their copies take, lock time, CPU, memory, I/O, or log volume. The metrics reference lists every metric and the labels that narrow it, such as a single table.
A test applies the bounds to the step it measures: a migration, a query, or a
Data Script. Each test below loads data first, and a step can add its own metric blocks
to the set:
- Migration
- Query
- Data Script
The test migrates to the previous version, loads data, and then applies the agent's migration:
test "migrate" "orders_synthetic" {
migrate {
to = "20261118090000"
}
dataset {
use = dataset.synthetic.orders
}
migrate {
to = "20261119120000"
expect "profile" {
use = [expect.profile.prod_safe]
}
}
}
The test runs a query over the loaded orders. Its own db.rows bound fails a query that reads the
whole table instead of the pending orders:
test "migrate" "pending_orders_synthetic" {
migrate {
to = "20261119120000"
}
dataset {
use = dataset.synthetic.orders
}
exec {
sql = "SELECT count(*) FROM orders WHERE status = 'pending'"
expect "profile" {
use = [expect.profile.prod_safe]
metric "db.rows" {
table = "orders"
operation = READ
le = 1000
}
}
}
}
The test runs the cancel_pending Data Script, which cancels the
pending orders:
test "migrate" "cancel_pending_synthetic" {
migrate {
to = "20261119120000"
}
dataset {
use = dataset.synthetic.orders
}
script "exec" {
file = "scripts.hcl"
run = "^cancel_pending$"
expect "profile" {
use = [expect.profile.prod_safe]
}
}
}
Schema tests take the same block on the same steps, so the functions, procedures, and triggers an agent writes get the same bounds. A bound Atlas can't verify fails the test too, so a pass means every bound held.
The bounds live in the repo, so put the file that holds them under code owners. An agent can change database code on its own. Loosening a bound takes a person.
Datasets, profiling, and profile expectations are in preview and open to a subset of Teams and Enterprise customers. To get access, reach out through your Atlas support channel or message us on Intercom.
Same Limits, Two Datasets
The bounds run twice before anything reaches production, each time on different data.
A failed bound in either run goes back to the agent. Only a change that holds every bound on real data deploys.
In the Agent's Sandbox
The sandbox test loads synthetic data. Atlas generates
rows from the schema, total sets how many rows the table holds in production, and a column block
sets the values a test depends on:
dataset "synthetic" "orders" {
table "orders" {
rows = 100000
total = 50000000
column "status" {
value = random::int(1, 1000) == 1 ? "pending" : "shipped"
}
column "region" {
value = random::element(["us", "eu", "apac"])
}
}
}
Atlas generates 100,000 orders, about one in a thousand of them pending and each with a region, and
checks lock times and table copies at 50 million. A column value can read the row's index and its
other columns, call random:: and HCL functions, or write NULL, as
Setting column values shows.
A statement whose cost grows with table size fails on the agent's machine, in seconds, with no production access. Full-table scans under a lock, table rewrites, and large index builds all fail here.
Before Deploy
CI runs the same bounds on a production snapshot:
dataset "rds_snapshot" "prod" {
src = "s3://backups/exports/prod-2026-10-13/appdb?region=us-east-1"
}
rds_snapshot loads an Amazon RDS snapshot export in foreign-key order. include and exclude pick
the tables, a sampler loads part of a table plus every row it
references, and masks hash or redact sensitive columns before any row
reaches the test database.
This run catches what scaling can't predict:
- Real rows that break the change, like a NULL under a new
NOT NULLor a duplicate under a new unique index. - Skew, like a backfill whose
WHEREmatches almost nothing in synthetic data and most of the real table. - Limits that kick in past a size. SQL Server escalates row locks to a table lock once a statement holds 5,000 of them, and a Postgres index build spills to disk once its sort outgrows memory.
Synthetic data is only as messy as you make it. Production data will always be messier.
The snapshot tests repeat the synthetic ones with use = dataset.rds_snapshot.prod in their dataset
step, such as orders_snapshot for orders_synthetic. The agent runs the synthetic tests on every
change, CI runs the snapshot tests before deploy, and only CI needs the credentials to read the
export:
# In the agent's sandbox
atlas migrate test --dev-url "docker://postgres/18/dev" --run synthetic .
# In CI, before deploy
atlas migrate test --dev-url "docker://postgres/18/dev" --run snapshot .
Keep those credentials in a CI environment that only the default branch can use. A test file can run programs, and a pull request can change it.
One Ticket, No Reviewer
Say the ticket is "make orders.region required," and the database is Postgres. Here's the loop with
no human in it.
Run 1. The agent writes the obvious migration:
ALTER TABLE orders ALTER COLUMN region SET NOT NULL;
The sandbox test fails on db.lock.time:
-- FAIL: orders_synthetic (1.02s)
migrate.test.hcl:23:
Error: db.lock.time: public.orders (AccessExclusiveLock, blocks reads):
~6.4s at 50,000,000 rows, expected less than or equal to 1s
(20261119120000_orders_region_not_null.sql:1)
Notice two things in the output:
- The test generated 100,000 rows and ran in about a second, but the lock time is for the 50,000,000 rows of production.
- It names the lock and points at the statement that took it, line 1 of the migration file.
SET NOT NULL scans every row to prove there are no NULLs, and it holds an ACCESS EXCLUSIVE lock
while it does. Every read and write on orders waits.
Run 2. The agent switches to the standard pattern: add the check as NOT VALID, validate it, then
set NOT NULL. Since Postgres 12, a valid check lets SET NOT NULL skip the scan.
ALTER TABLE orders ADD CONSTRAINT orders_region_not_null CHECK (region IS NOT NULL) NOT VALID;
ALTER TABLE orders VALIDATE CONSTRAINT orders_region_not_null;
ALTER TABLE orders ALTER COLUMN region SET NOT NULL;
ALTER TABLE orders DROP CONSTRAINT orders_region_not_null;
It fails on the same bound. Atlas runs each migration file in one transaction, and Postgres holds
locks until the transaction ends. The ACCESS EXCLUSIVE lock from the first statement is still held
while the second one scans the table, and Atlas counts that time too. Plenty of reviewers would
approve this file.
Run 3. The agent moves ADD CONSTRAINT into its own migration file. Validation now runs under a
lock that doesn't block reads or writes, and the sandbox test passes.
Run 4. CI runs the snapshot test, and VALIDATE CONSTRAINT fails. Orders created before the
column existed have no region, and the agent's synthetic data had a region on every row. Without this
run, the deploy would have failed halfway, after the first file had already shipped.
Run 5. The agent adds a backfill for those rows ahead of the validation, and the snapshot test passes.
Nobody reviewed any of it. Each failure named the statement and what broke, which is what an agent
needs to fix its own work. With --profile, it also gets the full report, which measures every
statement and flags the risky ones with a safer way to write them.
What the Numbers Don't Tell You
A test run measures the work your code does on your data, not the conditions it runs in. The work carries over: rows, rewrites, copy size, and log volume depend on the data and the statements, as long as the dev database runs your production version and settings. The conditions don't. A loaded snapshot has no live traffic, and a lock that takes half a second on an idle copy can take longer behind real queries, so leave headroom in time bounds.
CPU, memory, and I/O come from the Docker dev database, so they measure the machine that runs the test, the agent's sandbox or the CI runner. Use them to compare one change with another, not as production numbers.
The data has limits too. A sampled table can miss the one row that breaks a new constraint, so load
the full table for constraint changes. And the numbers at total assume synthetic rows behave like
real ones, which is what the snapshot run checks.
Wrapping Up
The factory doesn't have to stop at the database. Start with two
expectations, db.table.rewrites at zero and a
db.lock.time bound on your largest tables, and add more once you've seen your own numbers.
Expectations work on PostgreSQL, MySQL 8.0.28 and later, and SQL Server.
Profile expectations are in preview for a subset of Teams and Enterprise customers. To run them against your own migrations, schedule a demo or message us on Intercom for access.