debadyutirc

What entails the LLM Completion are you talking sequence of prompts with files / mcp servers. Could you share a bit more, cause I have spent some time with this and have something that might be precisely what you are asking for...

This is awesome. Love seeing more teams investing early in observability and evals instead of treating them as an afterthought.

Your setup (LLM-assessed complexity, semantic success metrics, tool-level telemetry) hits what a lot of orgs miss, tying evaluation and observability together. Most teams stop at traces and latency, but without semantic evals, you can’t really explain or improve behavior.

We’ve seen the same pattern across production agent systems: once you layer in LLM-as-judge evals, distributed tracing, and data quality signals, debugging turns from “black box” to “explainable system.” That’s when scaling becomes viable.

Would love to hear how you’re handling drift or regression detection across those metrics. With CoAgent, we’ve been exploring automated L2–L4 eval loops (semantic, behavioral, business-value levels) and it’s been eye-opening.

Fluvio is streaming transport. And we built Stateful DataFlow on top of that for Stream Processing.

Arroyo is SQL first stream processing. Fluvio is streaming transport which can send data to Arroyo and there is an integration.

Stateful DataFlow and Arroyo are similar in the stream processing pattern and the use of Apache Arrow.

The interfaces are different. Fluvio and Stateful DataFlow support for SQL is the same dialect as columnar SQL supported by Polars. The Fluvio and Stateful DataFlow paradigm is more intricate more expressive and the platform is broader and deeper.

We have had folks over the years asking us about the Kafka wire compatibility. We had a project 3 years ago which we archived. I think we have a case for reviving it in the near future.

This is a question we asked 6 years ago.

What if we wrote it in Rust. And leveraged and WASM.

We have been at it for the past 6 years. https://github.com/infinyon/fluvio

For the past 2 years we have also been building Flink using Rust and WASM. https://github.com/infinyon/stateful-dataflow-examples/

The search on HN is weird.

Isn't the Show HN: a different filter?

Agree with you 100%. We are working on more elaborate benchmarking on bare metal instances. This was just an initial run to utilize the benchmarking tool which is usable by all fluvio users.

We will do a full setup and benchmarks comparing Kafka, Pulsar, RedPanda using a real dataset on barmetal servers soon.

Really cool project! Look forward to trying this out. I have been using the copilot extensions with local docs toRAG augmentation. This seems to be a step up.

That's a really good suggestion. I am trying to build some tutorial videos now. I feel the same way about listicles but they get a lot of impressions. People seem to love lists.

Have been to a few pods. a few meetups.

Conferences has not been that great. Tried a couple.

The course option is very interesting. Have not tried something there.

Does it make sense to be part of some existing course? Or Should I create one?

Isn't it interesting that plays like the thing industries cant offer beyond QoS 0? Why do you say the gap is?

LOL. AI generated the image. But as far as I understand AJ wrote the rest. But it's a dry subject. I am curious what makes you say it is an AI generated Ad?

That sounds like an interesting approach.

I am curious about the details of how you do it.

No you don't. The open source project is here. You can self host and try it easy - https://www.fluvio.io/docs/fluvio/quickstart

Written by Wade Chambers, Chief Engineering Officer @ Amplitude

Does a great job at articulating the foundations of event driven architecture.

Would be interesting to see how this will grow into a full fledged application.

I work with the creators of Fluvio at InfinyOn.

Fluvio is an edge to core cloud native streaming engine built from the ground up in rust. Compiles to a single 37 Meg binary and deploys on ARM64 devices.

We just released the first public beta version of Stateful DataFlow. Stateful DataFlow is a framework for building unbounded distributed stream processing based on wasm that runs on Fluvio streams.

We are going for a Lean alternative to Kafka + Flink with a user experience of Ruby on Rails.

BTW, Stateful DataFlow has integrations with Arrow, Polars, and the ability to use SQL for dataframes, and other wasm compatible programming languages to express business logic. And Fluvio has Rust, Python, and JS clients.

Umm it's more complex than that. Open standard would still need money to survive. And Big Tech will flex their philanthropic muscle and influence open standards.

There is a lot of empirical evidence of this. Look at Matter protocol in home automation, GS1 in retail. Starts as a good idea but as soon as there is adoption, there will be several market motions to mess everything up.

The only way for an open standard to grow is a large extremely committed community that is largely made of people who don't have to worry about survival.

The app interface looks great. Great work.

The link does not open.

There are a lot of studies that support the rationale that blue light and apps are causing depression. It’s interesting that we are looking for solutions in apps as well.

I have got several BigTech interviews as well as jobs through regular posting long form posts on LinkedIn.

One of the best framings I have learned is to teach yourself by writing. Think of it as a conversation with your friends and you are learning together.

Take the negative comments as a motivation to improve.

Go for it.

This post prompted me to resurface the Genesis of Fluvio. We are a small team going at it for a little while since we have long relationship with the data centric applications across multiple domains for the past decades and we are excited about data streaming on rust and web assembly instead of java and jvm.

Here is a blog that is from June 2021 from our CTO laying out the vision for Fluvio.

https://news.ycombinator.com/item?id=38880743

Since the comparison question keeps popping up, I would share whatever we have as well. It's a long drawn documentation exercise and I am happy to share what we have at Fluvio.

Again - Great job with Iggy!

WOW this is a big release! I want to learn more about the EBS to S3 move. What are some of the constraints and how are you managing the latency with S3?

I lead product at Fluvio.

If I remember correctly, I had seen iggy.rs in one of the recent applicant resumes to one of our Senior Rust Developer roles. Iggy seems like a cool project. All the best for the future of Iggy.

Fluvio has been in open source development for nearly 5 years now. The team has written hundreds of thousands of lines of code.

Fluvio core is a complete distributed streaming system for unbounded stream processing that is reliable, scalable, self healing. It’s the Kafka bit.

The Flink bit is our Stateful Service Development Kit which is currently in Dev Preview. Stateful Service Development Kit offers a scaffolding to compose complex chained event driven data pipelines.

Once we launch the stateful materialization bit, we will have a complete system for building end to end streaming data flows.

Solid arguments. Interesting how they are going to deal with folks in different availability zones and geographies.

Yes. This is on the money.

Wasm component model enables the stream processing operations to be natively available to wasm compatible languages.

Fluvio core streaming engine is implemented in Rust.

The first generation stream processing, transformations were initially implemented in Rust needed binding generators to work in Python, JavaScript, Go etc. Also state and window operations and offset management needed to be expressed in application logic.

In the current generation of stream processing wasm component model enables a native support for polyglot stream processing.

Fluvio Stateful Service Development Kit is in Dev Preview. State operations, windows are part of SSDK.

A decent analog for Fluvio the core functionality of Kafka + Flink + Airflow. Distributed streaming + stateful stream processing + workflow orchestration.

Arroyo and Flink are great stream processing engines with SQL interfaces. Flink is more mature and Arroyo is fast and light since it is written in Rust.

The 2 key differences are: - The implementation of Fluvio SSDK is using the wasm component model, which natively enables polyglot interfaces. Fluvio operates using YAML to express schema, operators, state, and windows. - Fluvio SSDK provides a scaffolding for composing end to end event streaming flows in a single composable paradigm to reduce context switching and make the process of development simpler and uniform.

Thank you for the comment. This is certainly not a novel concept. Its an opinionated implementation that is geared towards solving some business problems.