HN user
https://turbolytics.io
Working on chain of custody for financial transactions!
Our first offering is OrderProof which records Shopify transaction evidence and generates PDF to help merchants streamline chargeback disputes.
https://turboops.io/products/orderproof
----
more about the problem here:
https://www.linkedin.com/pulse/you-losing-shopify-chargeback...
TurboOps.io - a workflow engine for humans!
https://turboops.io/platform/public-tracker
Our first offering is a tracker for makers, small businesses and contractors to show job status. Create a real time status page for your products, build trust and reduce customer inquiries.
We’re working on email notification support right now and have evidence (tracking numbers, job pictures, contracts/documents, etc) support coming next week!
Ha Yes! A pipeline assumes a "batch" of data, which is backed by an ephemeral duckdb in memory table. The goal is to provide SQL table semantics and implement pipelines in a way where the batch size can be toggled without a change to the pipeline logic.
The stream is achieved by the continuous flow of data from Kafka.
SQLFlow exposes a variable for batch size. Setting the batch size to 1 will make it so SQLFlow reads a kafka message, applies the processor SQL logic and then ensures it successfully commits the SQL results to the sink, one after another.
SQLFlow provides at least once delivery guarantees. It will only commit the source message once it successfully writes to the pipeline output (sink).
https://sql-flow.com/docs/operations/handling-errors
The batch table is just a convention which allows for seamless batch size configuration. If your throughput is low, or if you require message by message processing, SQLFlow can be toggled to a batch of 1. If you need higher throughput and can tolerate the latency, then the batch can be toggled higher.
Oh yes!! I've seen this a couple times. I am far from an expert in tributary so please take with a grain of salt.
Based on the tributary documentation, I understand that tributary embeds kafka consumers into duckdb. This makes duckdb the main process that you run to perform consumption. I think that this makes creating stream processing POCs very accessible. It looks like it is quite easy to start streaming data into duckdb. What I don't see is a full story around Devops, operations, testing, configuration as code etc.
SQLFlow is a service that embeds DuckDB as the storage and processing brains. Because of this, we're able to offer metrics, testing utilities, pipelines as code, and all the other DevOps utilities that are necessary to run a huge number of streaming instances 24x7. SQLFlow was created as a tool that I wish I had to for simple stream processing in production in high availability contexts :)
Stream to stream joins are NOT currently supported. This is a regularly requested feature, and I'll look at prioritizing it.
SQLFlow uses duckdb internally for windowing and stream state storage :), and I'll look at extending it to support stream / stream joins.
Could you describe a bit more about your use case? I'd really appreciate it if you could create an issue in the repo describing your use case and desired functionality a bit!
https://github.com/turbolytics/sql-flow/issues
We were looking at solving some of the simplier use cases first before branching out into these more complicated ones :)
Great Article! This is also timely for me, I spent all last week deep in the postgres documentation learning about replication (wish I had this article).
I'm building kafka connect (i.e. Debezium) compatible replication:
https://github.com/turbolytics/librarian/pull/41
-----
I really like the mongo change stream API. I think it elegantly hides a lot of the complexity of replication. As a side-effect of building postres -> kafka replication, I'm also trying to build a nicer replication primitive for postgres, that hides some of the underlying replication protocol complexity!
Still hacking on some data tools:
DuckDB for stream processing:
https://github.com/turbolytics/sql-flow
Lightweight kafka stream processing using DuckDB as the execution engine. 300MiB runtime can easily handle thousands of messages / second.
Working on a Kafka Connect alternative:
https://github.com/turbolytics/librarian
Right now mongo replication (through changestreams) is supported to kafka. Working on Postgres support right now.
How are you defining “B2C businesses on Main Street.”
The reason I’m wondering is because it’s striking how much more financially challenged my Main Street is compared to Vc texh.
I see so much opportunity for a small medium business consultation in the analytics and process space but these companies are like really strapped for money and largely set in their ways in my experience.
In my experience, people are open to solving their problems. It’s just the money is hard making it financially viable so it’s just the big money is just like an order of magnitude smaller.
Another thing I’ve noticed is that I think the general level of sort of like process thinking and data driven decision making in tech is at just like a higher baseline than on Main Street
A lot of my discussions are challenging how to sort of like present the problem in a way that somebody that doesn’t have decades of experience and operations understands.
Another challenge that I face regularly with Main Street companies is just people seem to be happy like they’re not trying to continuously optimize like I’m used to doing coming for a big tech. Even when it’s easy to present like positive ROI opportunities there’s just like a comfort with the way things are done and a lot of people seem just happy governed by their scaling factors in exchange for that that comfort.
TLDR; challenges are financial and mentality.
Productionizing duckdb :) I built a streaming tool around duckdb that allows for high performance stream processing and a rich connector ecosystem:
https://github.com/turbolytics/sql-flow
Building a company around a tool is hard. There's been some interest but streaming is kind of commoditized.
I'm taking everything I learned building it and working on a customer-facing security product, more to come on that :)
Built a stream processing engine using duckdb
https://github.com/turbolytics/sql-flow
It has some interest, unfortunately building tools as a business strategy is rough.
Beginning to work on first actual product! More soon :)
I just started using hurl a couple months ago.
For my uses it's great that it has both test suite mode and individual invocation mode. I use it to execute a test suite of HTTP requests against a service in CI.
I'm not a super big fan of the configuration language, the blocks are not intuitive and I found some lacking in the documentation assertions that are supported.
Overall the tool has been great, and has been extremely valuable.
I started using interface testing when working on POCs. I found this helps with LLM-assisted development. Tests are written to directly exercise the HTTP methods, it allows for fluidity and evolution of the implementations as the project is evolving.
I also found the separation of testing very helpful, and it further enforces the separation between interface and implementation. Before hurl, the tests I wrote would be written in the test framework of the language the service is written in. The hurl-based tests really help to enforce the "client" perspective. There is no backdoor data access or anything, just strict separation betwen interface, tests and implementation :)
Looked forward to move out day at state university in early 2000s. The university would rent dumpsters and place in the common outdoor areas. The dumpsters had the end door that would open so it was easy to walk inside of them without climbing.
I’d Spend all morning in the dumpster with some friends. Name brand clothes were good finds, also pretty much all the textbooks carried a trade in value. Lots of sealed food snacks as well.
I don’t know if the kids that threw them away were lazy or they just didn’t know about buy back, but the books easily brought me $100 for a couple hours of morning dumpster diving.
Me too :) - state university in early 2000s , would take the textbooks from the trash walk them over to the textbook exchange and pocket ~$100 for ~3 hours of work
Duckdb for streaming data!
https://github.com/turbolytics/sql-flow
I think the industry lacks lightweight fully featured stream processing solutions. I think it’s either heavyweight jvm or bespoke custom solutions
Sqlflow aims to be a middle ground , performant, fully featured, observable and supports sql
For the us, I feel like it’s late stage individualism. This is what happens I think when people prioritize themselves over their communities, I think we have less dependence on our communities than ever thanks to the internet and being able to physically avoid community. We have less interaction than before. We can order grocery pickup and not even have to be physically around people for basic life tasks. We order next day delivery on Amazon and don’t even have to go out in the world and be in the physical presence of others :(
The article talks about how it’s more of a younger generation phenomenon suggesting older generations still maintain their friendships
I’m grappling with this myself, it requires a lot of energy to form adult friendships. I keep seeing my neighbors out at the playground, I reach out and say hey and hi and ask them how they are doing but stop short of investing the time necessary to form real friendships with them and I know deep down that it’s perpetuating late stage individualism
Yes here too, and I’m guilty of the focus on family over friends, but also my peers are too and it creates a bunch of people that used to be close that never really see each other because they prioritize their immediate family
I def agree that there is a pattern to most data pipelines:
- read from an input (source)
- perform some sort of processing
- write the data to some output (sink)
This may either be batch or continuous (stream). The inputs may change, the outputs may change.
I personally think that sql and duckdb are well positioned to do this. SQL is declarative, standardized and has decades worth of mature implementations.
The “source” can be modeled as a table.
The “sink” can also be modeled as a table.
What does a custom dsl provide over sql?
I have a side project called Sqlflow which is attempting to do something similar/
https://github.com/turbolytics/sql-flow
It’s not a DSL but the pipeline is standardized using the source, process, sink stages. Right now the process is pure sql but the source and sink are declarative. SQL has so much prior art, linters and a huge ecosystem with many practitioners.
Congratulations, second streaming acquisition in recent memory, after benthos.
Arroyo on paper looks amazing, high performance, advanced streaming primitives like windows and joins.
I think / hope the foundations provide a long term alternative to flink and spark streaming, the extremely heavy jvm-based incumbents.
I only have ~5 hours of experience with it so def an ultra beginner, but it was very UI configuration heavy when I tried it. API based config was mentioned but didn’t slot into the way $dayjob provisions so it was a very hard sell.
In response to this I created a lightweight high performance dev-centric streaming solution powered by duckdb, it’s getting a little traction
https://github.com/turbolytics/sql-flow
Iceberg has been one of the most popular feature so far:
“The most common diagnosis associated with sudden cardiac arrest was coronary artery disease.”
I wonder the fitness levels of the people that experienced cardiac arrests. Like were these first time runners that signed up to run a marathon?
“That single gram of silica gel could have an internal surface area of eight hundred square meters—the size of almost two basketball courts.“
Can someone help explain this? I’m having a hard time wrapping my head around this. The tiny packet has this much surface area due to how much space exists within the balls???
In my experiences of living in suburbs for 30 years, I’ve seen the default is to ignore neighbors.
I don’t really get this. Our communities have so much in common and so much overlap, we shop at the same stores, go to the same parks, get stuck in the same traffic, our kids are at the same schools,our neighbors care for us medically, teach our kids, maintain our dwellings, work on our cars, and contribute to our local municipalities through property tax. We vacation at the same places.
We have so much in common but we put our heads down and duck into our homes ignoring our neighbors. To be honest it makes me really sick to think about. Like the internet has allowed us to live these parallel lives, highly dependent on our neighbors but completely isolated from them. We smile and nod then go to the ballots and kick our spite up to the federal level (in the US).
To me, we have the majority of our lives in common.
Social media and the political engines preys on our differences making them the focus of our interactions ignoring the fact that 90% of our day-to-day lives are overlapping and our concerns are similar: health, wealth, prosperity, safety, education and recreation.
It’s not much, but as I get older I’m making a point to slow down and talk to my neighbors, have real conversations with them, many of them fly political flags that are contrary to my political beliefs but I find out we have so much In common because we have such similar day-to-day lives and experiences.
Amazing! Thank you for the update, this will be huge
What’s the furthest a recipient will be forced to travel? How many offices per population area? What’s the current rate of calls and direct deposit modifications? At that rate what load will it put on the offices? How long is the average interaction? What will the throughput be if the current office staffing?
I've mentioend this whenever iceberg comes up. It's wild how immature the ecosystem is still. Duckdb itself lacks the ability to write iceberg....
https://duckdb.org/docs/stable/extensions/iceberg/overview.h...
Apache iceberg go ? Nope
https://github.com/apache/iceberg-go?tab=readme-ov-file#read...
Basically java iceberg is the only mature way to do this, it's not a very accessible ecosystem.
For a side project I'm using pyiceberg to sink streaming data to iceberg (using DuckDB as the stream processor):
https://sql-flow.com/docs/tutorials/iceberg-sink
It's basically a workaround for DuckDB's lack of native support. I am very happy with the Pyicerbg library as a user, It was very easy and the native Arrow support is a glimpse into the future. Arrow as an interchange format is quite amazing. Just open up the iceberg table and append Arrow dataframes to it!
https://github.com/turbolytics/sql-flow
Arrow is quite spectacular and it's cool to see the industry moving to standardize on it as a dataframe. For example, Clickhouse python also support arrow-based insertion:
https://sql-flow.com/docs/tutorials/clickhouse-sink
This makes the glue code trivial to sink into these different systems as long as arrow is used.
pfsh who needs to write data??? ;p
If you have streaming data as a source, I built a side project to write streaming data to s3 in iceberg format:
https://sql-flow.com/docs/tutorials/iceberg-sink
https://github.com/turbolytics/sql-flow
I realize it's not quite what you asked for but wanted to mention it. I'm surprised at lack of native iceberg write support in these tools.
Pyiceberg though was quite easy to use, arrow-based API was very helpful as well.
Interesting would love to learn more about their methodology.
Like are they straight searching “software developer” on indeed?
https://fred.stlouisfed.org/series/IHLIDXUSTPSOFTDEVE
—-
It’s rough out there, I see former coworkers laid off last year and the year before still struggling to find work
First of all I think this article was put together quite well, it was succinct had a great flow and was very easy to follow. The data was laid out to support the case; overall it was great to read.
Understanding a problem requires a holistic view of the larger system. An egg producer may say prices are higher Because supply is lower due to culling. Part of that statement is true. The author then zooms out to better understand the larger system.
The truth of a matter deviates significantly from one party’s assertions.
it’s scary to me that people lack the skills, the desire or incentives to understand and objectively seek understanding of what’s happening around us especially in highly charged US political landscape. This isn’t like pure scientific pursuit of knowledge but a more practical day to day ability to question, understand and gather new information with the purpose of developing an objective understanding of the world.
Fun, I did a similar project some years ago , I always like it when multiple people arrive at the same approach without ever communicating :)
https://github.com/dm03514/test-engine/blob/master/tutorials...
Very Cool! Which use cases are you targeting?
I'm surprised at the relatively sparse state of streaming. I feel like there are:
- Flink - Spark - ~Benthos - Arroyo - Then mostly custom / bespoke frameworks
Maybe there just isn't that much money in it? But I think there's still lots of opportunity to improve the Dev/UX over the JVM and the enterprise solutions.
-----
I would say recovery is achieved through kafka consumer groups right now, which results in at least once processing semantics.
We also support websocket input for the bluesky firehose but that is completely ephemeral. I have a story in the backlog to write ahead to disk which should allow for tolerating process crash/failure.
The tumbling windows stores state using duckdb. The end user can configure a disk-based duckdb databases which achieves durability as well.
The troubleshooting was actually pretty funky, i didn't realize how tricky it could be with stateful stream processing. I certainly don't have a good story for it, but what I did was added a sql debug HTTP handler. When this is enabled it exposes the duckdb execution context over HTTP. This was how I debugged the tumbling window logic during development. I would start SQLFlow, query to make sure the windows were empty, send some known messages, query the windows to make sure they were aggregated correctly, wait for the windows to close and flush, query the tumbling window state, etc.
For troubleshooting it also supports prometheus metrics oriented towards stream processing, # of messages, processing duration, success/failures, etc.
SQLFlow also ships with a dev framework, this allows you to execute your sqlflow pipeline against a json input file to make sure the sql processing logic is correct. I wanted to give the ability to decouple testing the logic, from actually having to stream data.
I would love to know about more what you're hacking on!