2 posts tagged with "schema-monitoring"

Automatic ER Diagrams and Docs for ClickHouse​

When working with a relational database like ClickHouse, understanding the database schema becomes essential for many functions in the organization. Who cares about the schema? Almost everyone who interacts with your data:

• Software engineers and architects use knowledge about the schema to make design decisions when building software.
• Data engineers need to have an accurate understanding of schemas to build correct and efficient data pipelines.
• Data analysts rely on familiarity with the schema to write accurate queries and derive meaningful insights.
• DevOps, SREs, and Production Engineers use schema information (especially recent changes to it) to triage database-related production issues.

Having clear, centralized documentation of your database's schema and its changes can be a valuable asset to foster efficient work and collaboration. Knowing this, many teams have developed some form of strategy to provide this kind of documentation:

• Diagramming tools. Teams use generic diagramming tools like Miro or Draw.io to maintain ER (Entity-Relation) Diagrams representing their database schema. While this is easy to set up, it requires manually updating the documents whenever something changes, often causing documents to go stale and become obsolete.
• Data modeling tools. Alternatively, teams use database modeling software like DataGrip or DBeaver. While these tools automatically inspect your database, understand its schema, and provide interactive diagrams, they have two main downsides: 1) Since they run locally, they require a direct connection and credentials introducing a potential security risk; 2) They do not enable any collaboration, discussion, or sharing of information.
• Enterprise Data Catalogs like Atlan or Alation, provide extensive schema documentation and monitoring; however, they can be quite pricey and difficult to set up.

Enter: Atlas Schema Monitoring​

Atlas offers an automated, secure, and cost-effective solution for monitoring and documenting your ClickHouse schema.

With Atlas, you can:

• Generate ER Diagrams: Visualize your database schema with up-to-date, easy-to-read diagrams.
• Create Searchable Code Docs: Enable your team to quickly find schema details and usage examples.
• Track Schema Changes: Keep a detailed changelog to understand what's changed and why.
• Receive Alerts: Get notified about unexpected or breaking changes to your schema.

All without granting everyone on your team direct access to your production database.

Getting Started​

Let's see how to set up Schema Monitoring for your ClickHouse database with Atlas. In this guide, we demonstrate how to run schema monitoring using a GitHub Action, but this can easily be achieved from other CI platforms (such as BitBucket or GitLab).

Prerequisites

1. A ClickHouse Cloud account with a running service.
2. An Atlas account (start your 30-day free trial here).
3. A GitHub repository with permissions to configure GitHub Actions workflows.

1. Create a bot token in Atlas Cloud​

Head over to your Atlas Cloud account and click on the top-level Monitoring navigation entry. Choose the GitHub Action card, and click on the Generate Token button. Copy the token.

Next, go to your GitHub repository and go to Settings -> Secrets and add a new secret called ATLAS_CLOUD_TOKEN with the value of the token you just copied.

2. Create a new GitHub Actions Workflow for schema monitoring​

Store your database URL as a repository secret named DB_URL with the value of your database url.

Next, save the workflow file below as .github/workflows/monitor-schema.yaml in your repository.

Replace the slug with the name you want to give to your database. The slug is used to uniquely identify the database in Atlas Cloud, even when the database URL changes.

name: Atlas Schema Monitoring
on:
 workflow_dispatch:
 schedule:
   - cron: '0 */4 * * *'  # every 4 hours
jobs:
 monitor:
   runs-on: ubuntu-latest
   steps:
     - uses: ariga/setup-atlas@v0
     - uses: ariga/atlas-action/monitor/schema@v1
       with:
         cloud-token: ${{ secrets.ATLAS_CLOUD_TOKEN }}
         url: ${{ secrets.DB_URL }}
         slug: my-database

Then, commit and push the changes to your repository.

3. Run the GitHub Action​

Once committed, let's run the workflow:

1. Go to the Actions tab in your repository
2. Choose the Atlas Schema Monitoring workflow
3. Click on Run Workflow on the top right corner.

After the workflow finishes, you should see a link to Atlas Cloud where you can view the schema of your database:

4. View the schema in the Atlas UI​

Click on the link provided in the logs to view the schema in the Atlas UI.

Amazing! We have set up continuous schema monitoring for our ClickHouse database using Atlas and GitHub Actions. The GitHub Action will run every 4 hours, ensuring that the schema documentation is always up-to-date, you can adjust the schedule to fit your needs or run the workflow manually on-demand.

Wrapping Up​

We hope you find this new integration useful! As always, we would love to hear your feedback and suggestions on our Discord server.

Additional Reading​

• Learn about using Drift Detection in tandem with Schema Monitoring.
• Run Schema Monitoring from a long-running Agent
• Schema Monitoring Documentation

Happy new year everyone, and welcome to our first release of 2025, Atlas v0.30! We have some exciting new features and improvements to share with you.

In this release you will find:

1. Simplified Schema Monitoring: Previously you needed to install a long-running agent on your database VPC to monitor your schema. Schema monitoring is now even simpler with the introduction of a new agentless monitoring mode.
2. Drizzle Support: We now support Drizzle, a popular ORM for Node.js. You can now use Atlas to automate schema migrations for your Drizzle projects.
3. Bitbucket Pipelines: We have added support for Bitbucket Pipelines, making it easier to integrate Atlas into your Bitbucket CI/CD workflows.
4. Custom Kubernetes Configurations: Atlas Pro users can now provide custom atlas.hcl configurations for their Kubernetes Custom Resource Definitions (CRDs) using the Atlas Operator.
5. txtar Editor Support: The Atlas JetBrains plugin now supports editing txtar files, used by the Atlas CLI to define pre-migration checks but also useful for other purposes.

Simplified Schema Monitoring​

We released Atlas Schema Monitoring a few months ago to enabled teams to track changes to the schema of any database. Previously, this workflow required installing a long-running agent process on your infrastructure.

To simplify things further, you can now run schema monitoring workflows directly from your GitHub Actions pipelines.

Learn how in this quickstart guide, or watch the video:

Drizzle Support​

Following popular demand from the Drizzle community, we are excited to announce our official guide on using Atlas to manage database schemas for Drizzle projects.

Drizzle already provides a powerful migration tool as part of the drizzle-kit CLI, which handles many schema management needs seamlessly. However, because it is deeply integrated with the Drizzle ORM, there are scenarios where a standalone schema management solution like Atlas might be a better fit.

In collaboration with the Drizzle team, we’re thrilled to highlight a brand-new feature introduced in drizzle-kit: the drizzle-kit export command. This feature, developed in partnership with our team, allows you to easily export your existing schema for use with Atlas.

To learn more, check out our Drizzle guide.

Bitbucket Pipes​

Atlas is designed to integrate seamlessly with your CI/CD workflows. Today we are excited to announce that we natively support Bitbucket Pipes, making it easier to automate your database schema management tasks with Atlas.

Using the new Bitbucket integration, users can easily perform schema management related tasks, such as running migrations, verifying schema changes, and monitoring schema drift, directly from their Bitbucket Pipelines.

For example, you can use the arigaio/atlas-action pipe to run migrations on your target database:

image: atlassian/default-image:3
pipelines:
  branches:
    master:
      - step:
          name: "Applies a migration directory to a target database"
          script:
            - name: "Migrate Apply"
              pipe: docker://arigaio/atlas-action:master
              variables:
                ATLAS_ACTION: "migrate/apply" # Required
                ATLAS_INPUT_URL: ${DATABASE_URL}
                ATLAS_INPUT_DIR: "file://migrations"
            - source .atlas-action/outputs.sh

To learn more, check out our Bitbucket Pipes guide.

Custom Kubernetes Configurations​

The Atlas Operator now supports custom configurations for your Kubernetes CRDs. This feature is available to Atlas Pro users and enables use cases like loading credentials from an external secret store or defining custom linting rules for verifying the safety of your schema changes.

To enable this feature, install the latest version of the Atlas Operator Helm chart using the allowCustomConfig flag. Then, provide your custom atlas.hcl configuration file in the spec.config field of your Atlas CRD:

apiVersion: db.atlasgo.io/v1alpha1
kind: AtlasSchema
metadata:
  name: sample
spec:
  envName: "test"
  schema:
    sql: |
      create table users (
        -- redacted for brevity
      );
  cloud:
    repo: atlas-operator
    tokenFrom:
      secretKeyRef:
        name: atlas-token
        key: ATLAS_TOKEN
  vars:
    - key: "db_url"
      valueFrom:
        secretKeyRef:
          name: schema-db-creds
          key: url
    - key: "dev_db_url"
      valueFrom:
        configMapKeyRef:
          name: schema-db-dev-creds
          key: url
  config: |
    variable "db_url" {
      type = string
    }
    variable "dev_db_url" {
      type = string
    }
    env "test" {
      url = var.db_url
      dev = var.dev_db_url
    }

For more information, check out the Atlas Operator documentation.

txtar Editor Support​

A few months ago, we shared our journey of building a robust testing framework for the Atlas CLI, in a blog post titled How Go Tests go test (originally created for GopherCon Israel 2024).

The post got a lot of attention in many Go communities (Including Golang Weekly #552) as it demonstrated how to utilize the Go team's internal tool for testing the Go toolchain itself.

Even more exciting, it seems that the post has stirred some interest in the Go community, with some fairly large projects adopting the same approach for their testing frameworks (see acceptance testing for the GitHub CLI and Cilium).

Over the past few years using this method for testing the Atlas CLI, our team has grown very fond of writing tests in txtar format. We found it to be a very expressive and concise way to define test cases, and it has become a core part of our testing strategy. Our appreciation for txtar has led us to adopt it in other parts of our tooling, including making it the way to define pre-migration checks in Atlas.

To make it easier for our users (and other teams using txtar for other purposes) to work with txtar files, we have added support for editing txtar files in the Atlas JetBrains plugin. This feature is available in the latest version of the plugin, which you can download from the JetBrains plugin repository.

In addition to txtar support, this release also includes support for HCL code formatting, making it easier than ever to write and maintain your Atlas schemas. To see both features in action, check out the video below:

More News and Updates​

• SOC2 Compliance. We have recently completed our SOC2 re-certification for the third year in a row. This certification demonstrates our commitment to providing a secure and reliable infrastructure for our users and customers. You can read more about this in our recent blogpost.
• Podcast Appearances. Catch Atlas on some recent pod cast episodes in: Kube.fm, The IaC Podcast, and Amazic.

Wrapping Up​

We hope you enjoy the new features and improvements. As always, we would love to hear your feedback and suggestions on our Discord server.