karakanb

İngestr is meant to be a lot more simpler to operate than Debezium and the likes of it, so yes, it makes things a lot easier.

About the staging dump, I haven't tried it but I don't see why it wouldn't work? I'll give it a try.

Yes, we'll create and share a helm chart, although we already publish Docker images so it should be trivial to spin up a chart if you wanted to.

One of the reasons why ZeroFS seems interesting is they use SlateDB under the hood, which optimizes the requests that hit S3 behind the scenes.

It is not obvious from the post but it seems like the allow list for the scripts supports whitelisting packages instead of a global setting. This should make it easier to maintain org-wise rules to allow scripts only for specific packages.

Is there a linter that could be used for scenarios like this to prevent unsafe default on package manager config?

Hi, this looks interesting, thanks for sharing. I am the builder of ingestr (https://github.com/bruin-data/ingestr), so I am very much in the same space.

I really like that you did this in Go, and I'll definitely dig a bit more into the source code to see how you tackled the CDC stuff, given that there is not many reliable CDC libraries in Go, and there are quite a few gotchas when it comes to doing CDC right. We also hand-rolled ours in ingestr, or I must say clanker-rolled, and we got quite a few things wrong in the first place.

Curious about the postgres-compatible query option: what's the usecase you have in mind there? My perception is that any org that would use Iceberg also has one or a few query engines in place, is this more for debugging stuff?

Quite cool stuff, keep it up!

Zot seems interesting, this is the first time I see it. On a quickl look it seems like Pi, but in Go. I was hoping to embed Pi into some of our internal projects and the typescript stuff was blocking me, I'll definitely give Zot a look.

That's a good point, thanks!

The project has been under development for over 6 months. We just open sourced it with a clean history. I am not sure what you expect here, should a project exist for months before it is worthy of a show post?

DAC currently supports YAML and JSX, what else would be a good alternative?

Thanks! DAC does that kind of validation partially, although doesn't validate the usage of the downstream dashboards. That's a very nice idea though.

In terms of validation it will validate queries, metric definitions, chart definitions and all ahead of time, before render. That way agents tend to validate their work much quicker.

I am not sure we are on the same page, as far as I am aware Vega doesn't do layout, does it? E.g DAC could use Vega for the charts and still take care of everything else around it.

There are quite a few libraries for charts and visualization, there are not as many for actually combining many of them with layouts, different components and including the actual implementation of the backend. Dac aims to provide all that as a standard and an implementation.

Point taken, thanks! Out of curiosity, which DAC did you come here for?

Partly, yes. It is a simplification with the perspective that the agents would be the primary builders.

You write a few lines of YAML or JSX and you get a dynamic, interactive dashboard out of it. Do you have any suggestions on how to make it simpler?

That's a great idea, will do very quickly, thanks!

Hi, author here, thanks! I have used TypeScript before across various projects, but I haven't considered building CLI tooling in that before, I guess due to my prejudice against the whole JS ecosystem. I plan to give it another try in the next weeks.

I have been recently looking into extracting a bunch of details from a set of legacy invoice PDFs and had a subpar experience. Gemini was the best among the ones that I tried, but even that missed quite a bit. I'll definitely give this a look.

It seems like such a crowded space and there are many tools doing document extraction, I wonder if there's anything particular pulling more attention into the space?

This seems to undermine the engineering muscle these companies have. Fivetran is well-capable of building a query engine, and with this merger, they also get access to SDF's query engine. They have the engineering capabilities, as well as the capital to attract the talent where needed.

I would not underestimate any of these players in the space.

OpenFlow feels like an attempt at keeping customers within Snowflake boundaries, and while it might work for some, I see no way Snowflake being able to keep up with the data integration needs unless they allow another way of extending their capabilities other than pre-built integrations.

On the other hand, I do agree with you that it is quite a big challenge for Fivetran to try and become Snowflake.

Disclaimer: I am the co-founder of a dbt alternative, Bruin. (https://github.com/bruin-data/bruin)

I think consolidation in the space has been coming for quite some time now and this merger only confirms what us, along with many others, have been saying: the data tooling is in a miserable state and we had to glue together a bunch of different tools that don't work with each other.

At this point, I think it is quite obvious that Fivetran is going for Snowflake/Databricks's market share. They own the ingestion for many companies already, and they will offer a managed data lake product in order to compete with the data giants. By owning the means of bringing the data in (Fivetran) as well as the transformation layer (dbt/sqlmesh) they will aim to get ahead of Snowflakes of the world.

I think it'll be a win for the data community if they maintain and continue investing into the existing tooling, as they are running in quite a few places already, especially dbt core running in a self-managed way. I certainly hope they won't try to squeeze revenue for the sake of it from their combined users.

It's an interesting time to be in the space, and it feels great to be one of the few independent players in the market.

This seems really interesting, thanks for sharing.

I can see myself deploying something like this for some of our internal usecases, however the deployment options around such tooling seem to be rather complicated, especially when it comes to the need for RAG and stuff. Would I need to host something else on top of Upsonic for these sorts of usecases?

while MySQL is very popular, it is very rare to see it in analytical/ML usecases, that's why we haven't added it yet. There's nothing from a technical POV that prevents us from adding, it just hasn't been a priority, I am happy to pull it up if that would help your usecases.

thanks! we already use DLT under the hood with ingestr, so some of our connectors already come from there. is that what you meant?

hey, thanks a ton for sharing your thoughts, I appreciate that!

I am sorry to hear that it didn't work, we do have a dedicated page for duckdb specifically here: https://bruin-data.github.io/bruin/platforms/duckdb.html

Would this help with it? I'd love to see how we can improve if you'd like to share your thoughts on that. Please feel free to join our slack community as well, we can talk directly there too.

hey, thanks a lot for sharing your thoughts.

I like the comparison page in Hamilton, and in their examples they operate in the asset level, whereas Bruin crosses the asset level into the orchestrator level as well, effectively bridging the gap there. What Bruin does is beyond a single asset that might be a group of functions, it is basically being able to build and run pipelines of that.

In terms of distributed execution, it is in our roadmap to support running distributed workloads as simple as possible, and Postgres as a pluggable queue backend is one of the options as well. Currently, Bruin is meant as a single-node CLI tool that will do the orchestration and the execution within the same machine.

love it, thanks!

glad to hear you like it, thanks!!

that's definitely coming, thanks!

hey, I didn't know that, definitely gonna take a look. thanks!

hey, absolutely. take a look at here: https://bruin-data.github.io/bruin/vscode-extension/overview...