woyten

Currently working on adding MySQL support to Greenmask (a database anonymization and synthetic data generation tool).

The interesting part is that it’s not just about adding another database driver. I’m revisiting a big part of the codebase and introducing a framework that should make implementing support for new DBMSs much simpler. The goal is to make Greenmask more extensible so that the community can add support for other databases without needing to dig through the entire internal architecture.

Published the first beta of this new approach a few months ago, and now the focus is on stabilizing it and making it production-ready.

MySQL support discussion: https://github.com/GreenmaskIO/greenmask/issues/222

Beta release: https://github.com/GreenmaskIO/greenmask/releases/tag/v1.0.0...

In case you don’t have FKs you could specify virtual foreign keys in Greenmask.

Check out docs for details

https://docs.greenmask.io/latest/database_subset/#virtual-re...

May I ask how often and in what situations you deal with data import tasks like these? This would help me better understand the issue and make the product more user-friendly. I would appreciate your input.

Hi! Thank you for the feedback. I completely understand your concern and agree with you. It can indeed be difficult to get started, and we need to provide clearer use cases with examples to demonstrate the basic concepts and features. I’ll be working on revising the documentation soon to make it easier to follow.

that a schema migration or code deployment would?

provide GUI and governance controls on top of that?

Exactly. Data doesn’t exist in isolation. Databases are dependencies of services, and schemas evolve throughout the software lifecycle, often managed by different data migration tools. In large organisations, regular developers usually don’t have direct access to the data sources, and masking rules along with real data sources are often restricted. Schema changes must be validated by the responsible data governance teams to ensure compliance and accuracy.

That’s why we implemented the validate command even in this standalone tool, which checks for schema differences and prevents running a dump if any schema changes are detected with detailed warnings. https://docs.greenmask.io/latest/commands/validate

I once presented Greenmask at an event organized by Percona in Cyprus, and one of the questions raised was: “What if we have a staging database, but instead of cleaning up the database and data, we want to add something to the existing dataset?” At the time, I didn’t have an immediate answer. However, this question inspired me to think, and eventually, I found a solution that at least partially covers this case:

You can restore data in topological order by preserving references and ensuring proper dependency handling (https://docs.greenmask.io/latest/commands/restore/#restorati...)

You can exclude non-critical errors to streamline the process without disrupting key operations (https://docs.greenmask.io/latest/configuration/#restoration-...)

I want to emphasize that this type of software must be flexible and adaptable to meet the ever-evolving needs of businesses… Otherwise, the project is as good as dead.

——————————

And I wonder if it would be worth talking to folks who do SOC 2 auditing

I’ve had discussions with professionals from Information Security, including those working in SOCs, and you're absolutely pointing in the right direction. At the moment, I’m actively exploring solutions and building a concept. I believe that by 2025, we’ll be able to showcase something new.

Thank you! I’m really happy that Greenmask meets your expectations

Hi! Thank you for your kind words.

The easier the tool is to use, the harder it might be to monetize... but there are so many applications of a tool like this

I am in favour with you! That’s exactly why we've started developing our Dynamic Staging Environments platform. It will integrate seamlessly with CI/CD systems, enabling to generate, create and maintain stateful components of your services more efficiently.

One key insight we’ve gained is that this type of software introduces new responsibilities across the Dev, Business, and Security teams within a company. This can result in complex processes and interactions. For example, changes to schemas often require approvals from both the security team and service owners, which can slow down workflows. That's why I really appreciate how Bytebase (https://www.bytebase.com/) tackled this issue—by providing a platform that combines GitOps for stateful resources with business logic in a streamlined way. We should be looking in this direction.

Important Remark:

Current design of Greenmask fits well with future plathform. We remain committed to focusing on simple utilities, as they are essential—especially for small businesses. These tools serve a critical purpose by solving problems quickly for smaller companies and for individuals who need a solution that works ASAP.

Hi! Noted. Alternatively, you can use the validate command, which will show you the transformation differences:

https://docs.greenmask.io/latest/commands/validate/

Thank you. Didn't know about that. I will use it for the next submissions.

Hello, community!

I am Vadim Voitenko, a database engineer and the developer of Greenmask (https://github.com/GreenmaskIO/greenmask), an open-source project for data anonymization in PostgreSQL. During my R&D work as a database engineer, I realized that database anonymization is not widely discussed or popular, despite many companies encountering issues in this area. Therefore, I decided to contribute and write a series of articles about data anonymization and the common problems people face. As a database engineer and enthusiast, I want to share the first article (https://hackernoon.com/database-anonymization-the-basics), which I hope lays the foundation for understanding the problem and provides examples.

Taking this opportunity, I would like to highlight several important features of Greenmask project that might be interesting for you:

* Validation: Powerful validation commands allow you to check transformation differences, constraint violations, and schema changes.

* Highly Customizable Transformation: For instance, customizable transformations such as random email generation (https://greenmask.io/v0.2beta1/built_in_transformers/standar...).

* Database Type Safety: Integrated with the pgx driver, ensuring accurate value encoding/decoding.

* Dynamic Parameters: Allows resolution of functional dependencies between columns (https://greenmask.io/v0.2beta1/built_in_transformers/dynamic...).

* Deterministic and Random Engines: Offers both deterministic and random transformation engines (https://greenmask.io/v0.2beta1/built_in_transformers/transfo...).

* Single Utility: Easy to set up and run.

* Schema Dumping and Restoration: Delegates these tasks to PostgreSQL's native utilities, as they are the most reliable.

* Backward Compatibility: Dumps created by Greenmask can be successfully restored by pg_restore.

* Large Objects Support: Handles the dumping and restoration of LargeObjects (https://www.postgresql.org/docs/current/largeobjects.html).

* Control Schema Changes: Checks your schema changes against the previous snapshot, prints the differences, and returns a non-zero exit code if changes are detected. This feature can be helpful in your CI/CD pipelines.

There is definitely a lot of work ahead. Have a look at the planned features in our Roadmap (https://github.com/orgs/GreenmaskIO/projects/6).

We have prepared a playground sandbox for experiments and for evaluating the Greenmask utility. You can access it here: (https://greenmask.io/v0.2beta1/).

If you want to run a Greenmask playground for the beta version execute:

git checkout tags/v0.2.0b1 -b v0.2.0b1

docker-compose run greenmask-from-source

I would appreciate feedback.

There is the latest release

https://github.com/GreenmaskIO/greenmask/releases/tag/v0.2.0...

Hi Greg! I am Vadim Voitenko - Greenmask [1] developer that you've mentioned in your message. Thank you so much for your interest.

[1] https://github.com/GreenmaskIO/greenmask

We are trying to deliver new features ASAP according to the user requests, if it not not a secret, could you list the features that you are looking for?

Since 5 days ago we have had a huge major beta release where we introduced important features - dynamic parameters and deterministic transformers and more. Check if there are features you need. If not so, I would love to hear them in order to prioritize them.