HN user
Data Engineer https://data.catering/ https://github.com/pflooky
I have a tool[1] that solely worked with docker before and was putting off supporting podman for a while because I thought it would take some time. But it turned out to work straight out of the box without tweaking. Essentially frictionless.
[1] Tool for reference: https://github.com/data-catering/insta-infra
It leverages docker compose 'depends_on' for the dependencies (https://docs.docker.com/compose/how-tos/startup-order/). For example, airflow depends on airflow-init container to be completed successfully which then depends on postgres.
https://github.com/data-catering/insta-infra/blob/main/cmd/i...
I have a small open-source project, that uses docker compose behind the scenes, to help startup any service. You can look to add it in (or I am also happy to add it in) and then users are one command away from running it (insta moose). Recently just added in lakekeeper and various data annotation tools.
insta-infra: https://github.com/data-catering/insta-infra
This looks really cool. I'm surprised I didn't find this before when I was searching for something like this. I've been using jpackage[1] for a while now but this seems like it would be easier for me to manage using JReleaser given there is support via Gradle.
Would this be a simple lift and shift job to move to JReleaser (as it seems like it just uses jpackage behind the scenes)? With jpackage, if you want to create a Windows exe, it needs to be built on Windows. Similarly, build dmg on Mac and deb for Linux. Does Jreleaser also require this?
[1] https://docs.oracle.com/en/java/javase/22/docs/specs/man/jpa...
I've taken a stab at making a solution for it via https://github.com/data-catering/data-caterer. It focuses on making integration tests easier by generating data across batch and real-time data sources, whilst maintaining any relationships across the datasets. You can automatically set it to pick up the schema definition from the metadata in your database to generate data for it. Once your app/job/data consumer(s) use the data, you can run data validations to ensure it runs as expected. Then you can clean up the data at the end (including data pushed to downstream data sources) if run in a shared test environment or locally. All of this runs within 60 seconds.
It also gives you the option of running other types of tests such as load/performance/stress testing via generating larger amounts of data.
This was also my philosophy behind creating insta-infra (https://github.com/data-catering/insta-infra). Single command to run any service. No additional thinking required.
Too many times I've become very frustrated when an installation doesn't work the first time or it has some dependencies that you haven't installed (or worse, you have a different version). Then you end up in some deep rabbit hole that you can't dig out from. Now for each tool I make, it must have a quick start with a single command.
The world of mock data generation is now flooded with ML/AI solutions generating data but this is a solution that understands it is better to generate metadata to help guide the data generation. I found this was the case given the former solutions rely on production data, retraining, slow speed, huge resources, no guarantee about leaking sensitive data and its inability to retain referential integrity.
As mentioned in the article, I think there is a lot of potential in this area for improvement. I've been working on a tool called Data Caterer (https://github.com/data-catering/data-caterer) which is a metadata-driven data generator that also can validate based on the generated data. Then you have full end-to-end testing using a single tool. There are also other metadata sources that can help drive these kinds of tools outside of using LLMs (i.e. data catalogs, data quality).
I recently went down the rabbit hole of using PyScript for running a Python CLI app in the browser.
It felt hacky the whole time, especially when dependencies were involved. I had to create wrapper classes to work around Pydantic 2.x not being available to use. I tried to put all logic into the Python files but found some things missing that I had to put in JavaScript.
I think it could be good in use cases where you want some simple UI with custom UI logic on top of your Python code but maybe Streamlit or Gradio could be more suitable.
GitHub repo: https://github.com/data-catering/data-contract-playground
Website: https://data-catering.github.io/data-contract-playground/
I've created a Docker image for it and onboarded it into my tool called insta-infra[1]. You should be able to run it via:
./run.sh maestro
Would love to hear what people think or other approaches people have taken to help quickly spin up tools on your laptop.
I've been keeping an eye on these kinds of Spark accelerator libraries for a while now.
How does it compare to Blaze[1] and Gluten[2]?
I'm interested in running some benchmarks soon against all three for my project to see how they all go.
Thanks for sharing. Happy to see another solution that doesn't just slap on AI/ML to try to solve it.
I am also among the many people who have created a solution similar[0] to this :). The approach I took though is being metadata-driven (given most anonymisation solutions cannot guarantee sensitive data not leaking and also open up network access from prod to test envs, security teams did not accept it whilst I was working at a bank), offering the option to validate based on the generated data (i.e. check if your service or job has consumed the data correctly) and ability to clean up the generated or consumed data.
Being metadata-driven opened up the possibility of linking to existing metadata services like data catalogs (OpenMetadata, Amundsen), data quality (Great Expectations, Soda), specification files (OpenAPI/Swagger), etc., which are often underutilized.
The other part that I found whilst building and getting feedback from customers, was having referential integrity across data sources. For example, account create events coming through Kafka, consumed and stored in Postgres whilst, at the end of the day, a CSV file of the same accounts would also be consumed by a job.
I'm wondering if you have come across similar thoughts or feedback from your users?
Working on a data generation and validation tool called Data Caterer. The focus of it is being data source agnostic, fast and simple. Just last week, I released a UI for it.
I think they make the biggest difference when testing data pipelines (which have historically been difficult to test). You can now easily test out compatibility between different versions of databases, verify data types, embed as part of your build, etc.
I believe the next step, once using test containers, would be automating data generation and validation. Then you will have an automated pipeline of integration tests that are independent, fast and reliable.
This is what I am trying to solve via building Data Catering (https://data.catering/). It gives you the ability to generate data into any database (along with maintaining any relationships between data) via metadata that can be retrieved via a source database or other types of metadata sources (i.e. Open metadata).
Interesting feature. One key thing I found when testing is that for you to reproduce the set of steps the user went through, there are a data attribute(s) that need to remain the same. For example, after login, a request for your account information will contain an account_id number that should be the same for all other account requests. If you can't guarantee this, then I don't see how you could use this in any sort of integration tests.
Isn't it simpler to use the Open API spec then generate from there?
Grats on building this out. I think there is a lot of potential in this space. I very much understand the challenges of financial/regulatory reporting and data quality :).
Couple of things I have noticed. You mention "automates root cause analysis". By this I assume you mean showing which rows have affected the metrics to go out of bounds? Or is there something else I'm missing.
How do users define metrics? I've found this to be a challenge especially given you may be giving this tool to non-technical users.
Does this support real time data sources such as Kafka?
Do you plan on supporting cross dataset validations (i.e. relationships such as an account_id for a transaction should exist in the accounts table)?
The thoughts here are a bit too focused on the costs of testing. Although deeper questions are asked about your own judgement of whether to test something or not, it doesn't consider the benefits. I wrote about this the other day in another post:
- Test give the ability of other developers to be productive on the project faster. Having tests tells other developers the intended behaviour and notifies them when they have broken it.
- Flow on effect of slow or large test suites. It demotivates developers to write new tests, run existing tests and accept failures if there are too many. This then leads to a lack of confidence, trust deteriorates between team members and development pushes towards going faster than being stable. Eventually, once enough bugs occur or a large incident happens, then you go back to looking at tests again.
If you have fast and reliable test suites, a developer wants to run and add to them. Developers feel that this is a high quality project that needs to be well maintained. This culture then permeates into other areas of your business.
(Sorry for the self plug) This is why I created Data Caterer (https://github.com/data-catering/data-caterer) to try provide a fast and reliable tool to help with end to end testing.
It was touched on at the beginnning of the article but something I find overlooked about having tests are:
- Ability of other developers to be productive on the project. Having tests tells other developers the intended behaviour and notifies them when they have broken it.
- Flow on effect of slow or large test suites. It demotivates developers to write new tests, run existing tests and accept failures if there are too many. This then leads to a lack of confidence, trust deteriorates between team members and development pushes towards going faster than being stable. Eventually, once enough bugs occur or a large incident, then you go back to looking at tests again.
If you have fast and reliable test suites, a developer wants to run and add to them. Developers feel that this is a high quality project that needs to be well maintained. This culture them permeates into other areas of your business.
Another side effect to take into consideration when offering notifications to customers is the impact on your services if customers action on those notifications.
At a bank I worked at, we enabled salary transaction notifications. Soon we found that because the salary was deposited into everyone's account at the same time from the bank, a large amount of notifications was trying to be sent at the same time which caused a bunch of services to suddenly receive a lot more traffic than usual.
I was using the following RFC for the formal definition of CSV https://www.rfc-editor.org/rfc/rfc4180.html.
When the delimiter is different, you have a different file format. TSV (tab-separated values) is an example of this.
Good in-depth insights into each format. This complements nicely with a site I created called tech-diff (https://tech-diff.com/file/) where it provides a summary of the file formats.
I also had a similar idea when I built tech-diff (https://tech-diff.com/) but rather than being just a catalog of items, it shows what the feature differences are. This is what I would argue is more important when investigating alternatives.
But, this comes at the cost of the data being harder to obtain as it is usually deep within documentation or code. Maybe we can reach a feature specification of technologies sometime in the future.
Thanks for sharing. Looks simple and easy to use.
Could you use this for batch data jobs as well? I would imagine having integration with batch job frameworks like Spark would make this more valuable from an organisation perspective.
Also, small note on the website, as an example from the "What is streamdal" page, I feel like I'm bombarded with emojis, bold and italic text, links, etc.
Agreed. I guess it depends on the target market. If your target is small to medium sized businesses, this is great to start doing analytics. But for large organisations, they generally have ETL/ELT jobs that do all the extraction from data sources to push to an analytics store or save in some format that is performant for analytics and storage (i.e. parquet). I'm also not sure how many data visualisation tools support API endpoints for query serving.
Thanks for sharing. Looks very clean and simple to use.
Do you plan on supporting non-JSON data types for insertion? For example, inserting CSV files, parquet files, Avro or Protobuf messages?
This looks very neat! I like the approach of including all data (API, messaging, database, files) as I went through similar thoughts and implementation recently with my project.
A growing pain for data integrations is data validations. The schema can be correct but the data itself may have incorrect values. I noticed you had a section about the structure and semantics of the data here (https://orbitalhq.com/docs/background/intro-to-semantic-inte...) but don't see where you could validate records using it. How do you handle these scenarios?
Very neat! Thanks for sharing. Have you thought about how you could export out all of these validations? For example, these validations/contracts can be used as metadata for data consumers who can know and trust your data based on the validations/contracts.
Thanks for the response. I also noticed there was a mention of data contracts or Pydantic to keep your data clean. Would it make sense to embed that as part of a DLT pipeline or is the recommendation to include it as part of the transformation step?
Do you plan on integrating with metadata sources as well such as Amundsen or Datahub? Or is the plan that DLT will become the metadata source?