a8m

We just shipped Atlas v1.1 - The focus of this release is Database Security as Code and Declarative Data Management.

You can now declare database users, roles, permissions, and managed seed data as code - versioned, reviewed, and applied via CI/CD like schema changes.

Feel free to share any thoughts or feedback.

Ariel from Atlas here. This setup is pretty common (declarative locally, versioned in real environments). Since migrations are auto-generated (by Atlas) on PRs, most developers never touch the versioned workflow directly.

See: https://atlasgo.io/concepts/declarative-vs-versioned#combini..., and https://github.com/ariga/atlas-action?tab=readme-ov-file#ari...

Good point regarding ORMs - that was one of the main problems I wanted to tackle when we built Atlas (https://atlasgo.io). We added support for reading ORM definitions directly, then let you extend the "base schema" defined in them. For example, you can define your models in SQLAlchemy, EF Core, Ent, or others as a partial schema, and then extend it with functions, views, and additional objects.

From there, Atlas handles diffing, planning, and execution. This is similar to importing modules in TF, but for database schemas in Atlas. See this example: https://atlasgo.io/guides/orms/sqlalchemy

Disclaimer: I'm involved with Atlas.

I'm glad to see I'm not the ~only one bothered by this issue. This has been, and still is, one of the challenges in managing MySQL schemas.

That's one of the reasons we implemented "migrate down" differently than other tools.

I'm not here to promote my blog post, but if you are interested in seeing how we tackled this in Atlas, you can read more here: https://atlasgo.io/blog/2024/04/01/migrate-down

Check out: https://github.com/ariga/atlas. It’s like Terraform for databases.

(I'm one of the authors of this project).

Hey! For a long time, one of the most common feature requests we've been getting from our users is the ability to manage their desired "schema state" using SQL. For this reason, we're excited to announce the release of Atlas v0.9.0, which fully supports SQL.

We also released the Atlas Playground, where users can visualize their database schemas and plan migrations in an interactive way. Check out the example below and let me know what you think <3

Playground example - https://gh.ariga.cloud/explore/01d2aeb1

Hey, I am the creator of entgo.io. I'm sorry to hear that your experience with the project was not great. I try to be available on our Discord, Slack, or the issue tracker, but sometimes I do miss some messages. If you ever consider to rejoin the community, please don't hesitate to drop me a message on our Discord.

I would also like to emphasize that Ent is an open-source project under the Linux Foundation, and it is not monetized in any way. Most of my work on it in the last years was on my free time, including when I was working at Facebook and wanted to OSS it. We (heavily) use Ent in our own products today, and that is why we continue to sponsor and maintain the project. I love it and I find it useful.

Thanks for the honest feedback. I appreciate it.

Just to emphasize, that OSS version of Ent models SQL tables in the traditional way (edges as foreign-keys and join-tables). i.e., the database structure is not obscure or something and can be easily read by developers and ported to other ORMs in case it's needed.

Yes. The migration is built on top of Atlas: https://github.com/ariga/atlas

Happy to share with you the new version of Atlas, the terraform for databases and a migration engine.

There are many great features in this release, but one of the biggest features we added is multi-tenant support. Using this option, users can define, in a few lines of code, that a migration should be executed on N databases/environments.

Please, checkout the code examples in the link and share your thoughts on it. Your feedback is greatly appreciated.

Hey! Would love to share with you the latest release of Atlas - the migration engine we are working on.

Currently, this tool provides a variety of capabilities for versioned migrations, including automatic migration authoring, linting migrations, and their execution on the database. We are already working on features that will make troubleshooting and fixing bad migrations much easier.

Appreciate any feedback or suggestions you may have for improvements. Thanks <3

Views are going to be supported in the SQL-DDL version (see my comments below). I don't have any info about stored-procedures/triggers/events atm.

I agree with that as well. The idea is to create an infrastructure for SQL parsers. Base parser will hold all standard structure and dialects can register custom clauses/statements. At the moment, I generate PEG files for each dialect, but this creates too much duplicate code, and does not allow sharing same types/objects between different dialects.

I thought about keeping it on the same GitHub repository (https://github.com/ariga/atlas), but as a separate Go module? WDYT?

Thanks for the feedback. I'm one of the atlas's creator.

At the moment, you can define schemas using Go (with a fluent API) or with HCL. The reason we decided to use HCL is because it can be easily extended, and our plans are to allow attaching metadata/annotations to schema objects (like k8s annotations) - more details in future versions.

Having said that, we understand we can't cover all features of every database in HCL, and that's why we work on allowing users to define their schemas using SQL DDLs (e.g. "CREATE TABLE", "CREATE INDEX", etc).

The apply/plan output is SQL, and we don't have plans to change it. However, the core engine is already smart enough to generate you a "reverse" command and tell you if a migration is "reversible" and "transactional". The next (minor) version is going to introduce "migration authoring". A way to generate a migration output to a directory and integrate it with tools like flyway or go-migrate, and also gives you suggestions to break migration to multiple steps if it's not transactional or reversible (like a DBA).

Hey, I'm one of the atlas's creator. Thanks for the feedback.

I'm actually familiar with all the things you mentioned here (I worked at FB too ;)), and some of them are the reasons why we decided to create atlas and OSS it.

First, atm, we support HCL and Go (with fluent API) for describing schemas, but in the next versions, we'll add support for SQL DDLs (e.g. "CREATE TABLE", "CREATE INDEX", etc). Can't promise time estimation because it's in development, but that means we plan to OSS with it an SQL-parser infrastructure for the supported databases (can elaborate on that if you want).

Before I continue to migration authoring, I want to mention the reason we chose HCL (or Go). In next versions, we plan to support attaching "annotations" to schemas, like in k8s or in ent [1]. We these annotations, you'll be able to define privacy policy, or create integration to other tool. More details in the near future.

Now, migration authoring. The CLI does not expose all functionalities that are covered by the core engine atm, but when you run this tool (apply/plan) the output is a list of SQL statements. The core engine already knows to generate the "reverse" command for each statement (if it is reversible), and also a summary that indicates if the migration is "transactional" and "reversible" (see example [2]). Next version of atlas is going to support "migration authoring" - that means, instead of generate you list of statements and execute them (after approve), we'll let you the option to generate them to a directory, edit them, and integrate them with tools like flyway, go-migrate, etc.

In addition to that, the engine is also going to suggest you to break a migration plan to multiple steps (like a DBA) in order to make it transactional or reversible if it is not.

[1] https://entgo.io

[2] https://github.com/ariga/atlas/blob/master/sql/postgres/migr...

Thanks for the blog-post Bodo! It's really pleasant to read.