Show HN: Bytewax – Python Stateful Stream Processing on Timely Dataflow

by amath • 4 years ago

21 13 4 years ago

shruts02• 4 years ago

Pretty cool! Have you added anything to Timely Dataflow yet or is this just Python bindings?

amath5• 4 years ago

Zander here, part of the team working on Bytewax. Much of the magic of Timely is passed through to the Python library, but there are a few features that we thought would make it easier to build with and run in production.

- We deepened the integration between inputs and output sources starting with Kafka/Redpanda. - We have added state recovery backed by either SQLite or by a Kafka API compatible platform like Redpanda or Kafka. - Built a deployment tool called Waxctl that makes it easier to run dataflows across pods on a kuberentes cluster and deploy to remote compute cloud VMs like EC2. - Provide some helpers for things like windowing and ordering inputs. These are still evolving

We are always looking for feedback and for which features to focus on next!

mohinish• 4 years ago

This is amazing, would love to chat and connect to learn more on this.

TheIronYuppie• 4 years ago

Does this work on a single node only, or does it work across a cluster like Kubernetes?

amath5• 4 years ago

That’s one thing we are really excited about! Kubernetes makes it easy to scale Bytewax dataflows across multiple nodes relatively easily. We have some long format docs about how it works https://docs.bytewax.io/deployment/k8s-ecosystem.

mloncode• 4 years ago

Bytewax is the first streaming data python client that I can finally get my head around. I've used Bytewax and I love the developer experience, you folks really dialed it in there.

Also the docs are great, which many developer tools tend to neglect. Good docs are the first thing I notice when looking at a project.

Great work, and congrats on the launch.

amath5• 4 years ago

Thank you!

agallego• 4 years ago

I think this is a good architecture to focus on the developer experience. A rust runtime with a Python front end. Very cool to see

willempienaar• 4 years ago

Been following this project for a while. There's a reason why it's making waves in developer circles - You get a simple to use Python developer experience with a powerful distributed data processing framework that scales to enormous workloads. Plus its battle tested since it relies on Timely Dataflow under the hood.

caravel• 4 years ago

This is much more straightforward than your JVM-based, huge infra solutions like Flink. For when you need something that is familiar [python] and just works.

m6c6• 4 years ago

This looks cool bc you get the best of both worlds with a good developer experience in Python, but the performance of Rust. Are there any examples of scaling it on Kubernetes, that'd be an interesting evaluation.

Are you open to contributors btw? I'm interested in learning more about Python <> Rust bindings.

amath5• 4 years ago

Yes, open to contributors!

I would say checkout PyO3 to learn more about how you can marry Python with Rust code. It is an awesome project.

Show HN: Bytewax – Python Stateful Stream Processing on Timely Dataflow

Related Stories