Why Metaflow?

https://docs.metaflow.org/introduction/why-metaflow

by savin-goyal • 11 months ago

15 6 11 months ago

docs.metaflow.org

What do people do to curate/version /transform their raw datasets these days? I am vaguely aware of the "chuck it all into s3" strategy for hanging onto raw data, and related strategies where instead of s3 it's a db of some flavor. What are folks doing for record-keeping for what today's raw data contains vs tomorrow's?

And the next step - a curated dataset has a time-bound provenance - what are folks doing to keep track of the transformations/cleaning steps that makes the raw data useful for the data at the time it's being processed? Does this bit fall under the purview of metaflow, or is this different tooling?

Or maybe my assumptions are off base! Curious about what other teams are doing with their datasets.

patcon• 11 months ago

I'm exploring kedro and Kedro-viz lately, in case that's in the vicinity of your question. It ties most closely with MLFlow for artifacts, but storing locally works fine too

mastazi• 11 months ago

I think that this would be a better link as it actually explains what Metaflow is without too much marketing fluff https://docs.metaflow.org/introduction/what-is-metaflow

KaiserPro• 11 months ago

It seems like a half-remebered clone of facebook's flow: https://docs.metaflow.org/metaflow/basics

The plus side is that its not obviously async which is nice, the downside is that the syntax is less fun (you need to explicitly link forwards, rather than backwards.)

marksimi• 11 months ago

Curious to hear from folks who have used both Metaflow and Kubeflow to understand some of those tradeoffs.

Seems like Metaflow is comparatively lightweight, bit more tightly integrated with AWS, less end to end and a bit more agile.

ghilston• 11 months ago

I was a big fan of Metaflow a few years back. I thought it was neat how I could write some code and easily run some functions locally versus remote.

Hey Savin, it's been a while since chatted. I hope things are going well ;)

For those unaware, onsone of the co

Why Metaflow?

Related Stories