HN user
Helping companies build real-world ML/AI systems.
CEO/co-founder/customer support representative at https://outerbounds.com
I started https://metaflow.org at Netflix where I led ML infra for 3+ years. I wrote a book about my learnings:
https://www.manning.com/books/effective-data-science-infrastructure
Earlier projects:
http://traildb.io
http://tuulos.github.io/sf-python-meetup-sep-2013/
http://discoproject.org
If you are ok with executing your SFN steps on AWS Batch, Metaflow should do the job well. It's pretty inhuman to interact with SFN directly.
One feature that's in our roadmap is the ability to define DAG fully programmatically, maybe through configs, so you will be able to have a custom representation -> SFN JSON, just using Metaflow as a compiler
Stay tuned! We have some cool new features coming soon to support agentic workloads (teaser: https://github.com/Netflix/metaflow/pull/2473)
If you are curious, join the Metaflow Slack at http://slack.outerbounds.co and start a thread on #ask-metaflow
Metaflow tracks all artifacts and allows you to build dashboards with them, so there’s no need to use MLFlow per se. There’s a Metaflow integration in Weights and Biases, CometML etc, if you want pretty off-the-shelf dashboards
Metaflow was started to address the needs of ML/AI projects whereas Airflow and Dagster started in data engineering.
Consequently, a major part of Metaflow focuses on facilitating easy and efficient access to (large scale) compute - including dependency management - and local experimentation, which is out of scope for Airflow and Dagster.
Metaflow has basic support for dbt and companies use it increasingly to power data engineering as AI is eating the world, but if you just need an orchestrator for ETL pipelines, Dagster is a great choice
If you are curious to hear how companies navigate the question of Airflow vs Metaflow, see e.g this recent talk by Flexport https://youtu.be/e92eXfvaxU0
I don't know if it's a coincidence but we just released a major new feature in Metaflow a few days ago - composing flows with custom decorators: https://docs.metaflow.org/metaflow/composing-flows/introduct...
A big deal is that they get packaged automatically for remote execution. And you can attach them on the command line without touching code, which makes it easy to build pipelines with pluggable functionality - think e.g. switching an LLM provider on the fly.
If you haven't looked into Metaflow recently, configuration management is another big feature that was contributed by the team at Netflix: https://netflixtechblog.com/introducing-configurable-metaflo...
Many folks love the new native support for uv too: https://docs.metaflow.org/scaling/dependencies/uv
I'm happy to answer any questions here
if you want to see similar tricks applied in Python (with a JIT compiler for query-time optimization), take a look at this fun deck that I presented a long time ago: https://tuulos.github.io/sf-python-meetup-sep-2013
we were able to handle trillion+ datapoints with relatively modest machines - definitely a useful approach if you are ready to do some bit twiddling
yes, this. In case you are interested in seeing some numbers backing this claim, see here https://outerbounds.com/blog/metaflow-fast-data
Source: I used to work at Netflix, building systems that pull TBs from S3 hourly
I had the same concern. However, the structure of the output was surprisingly stable. We rejected badly formatted responses: https://github.com/outerbounds/hacker-news-sentiment/blob/ma...
The semantics of the topics/tags could be improved for sure with a more detailed prompt
here's how the model ranks the discussion on this page after 40 comments:
SENTIMENT 6
:D
yep! And of course the new 3.1 model
even simpler, you can just do it in SQL
You can find all titles and dates since the beginning of HN in this public BigQuery dataset: https://console.cloud.google.com/marketplace/product/y-combi...
That's an interesting hypothesis but the words we use to express agreement and disagreement haven't changed much.
We don't try to retrieve articles/topics from the model, which would be affected by the cutoff, just asking it to analyze the sentiment or summarize the content provided in a prompt
yes please! The data is conveniently available as JSON blobs here https://github.com/outerbounds/hacker-news-sentiment/tree/ma...
search "divisive" here: https://github.com/outerbounds/hacker-news-sentiment/blob/ma...
I actually spent 10 minutes trying to see if there are obvious tests for U-shaped distributions. I'd love to hear if anyone has ideas here.
Exactly. Now we have a new UI paradigm and a new, incredibly powerful NLP tool in our toolkit.
Is this AGI? Of course not. Is this useful and valuable? Obviously.
here's how we solved engineering challenges related to RAG using open-source Metaflow: https://outerbounds.com/blog/retrieval-augmented-generation/
Metaflow integrates with AWS Batch which many folks use for serious HPC. Internode scheduling happens through the multinode scheduling supported by AWS Batch. networking via EFA etc.
We'll blog more about this soon but you can certainly give it a try today! https://github.com/outerbounds/metaflow-ray
Congrats for the launch Stefan and Elijah! :)
Like Stefan mention in the OP, Hamilton works well with tools like Metaflow which can help with many other concerns you mentioned. How you define your data transformations for ML is an open question that Hamilton addresses neatly.
See here for an example of Metaflow+Hamilton in action: https://outerbounds.com/blog/developing-scalable-feature-eng...
Outerbounds | Systems / Full-Stack engineer | SF / Remote | Full-time
Outerbounds commercializes Metaflow, a widely used open-source Python framework that makes infrastructure easily accessible for machine learning/data science projects. Metaflow was originally started by us at Netflix and it is now used by hundreds of companies across industries.
We care a lot about thoughtful design, overall product experience, and quality of code. We are looking for backend/full-stack/systems engineers who are experienced in at least one of the following areas: Delightful API design, deep Python experience, distributed systems, Kubernetes, or low-level systems programming.
- Outerbounds: https://outerbounds.com
- Metaflow: https://docs.metaflow.org
- More about us here: https://outerbounds.com/workwithus
- Join our Slack to see the project in action: http://slack.outerbounds.co
- Email: workwithus@outerbounds.co
Totally! One intended use case of this integration is that it allows you to move easily to a more scalable orchestrator if you hit limits of Airflow
This is great! If you are interested in running ML workloads across clouds, Netflix's Metaflow will officially announce support for all clouds tomorrow: https://outerbounds.com/blog/metaflow-on-all-major-clouds/
Quite a coincidence :)
If you want to experiment with something similar by yourself and you don't have the patience to wait for Stable Diffusion to crunch through thousands of images on your laptop or in a Colab notebook, here's how you can parallelize processing relatively easily on AWS Batch or Kubernetes: https://outerbounds.com/blog/parallelizing-stable-diffusion-...
Thanks for sharing! I am still proud of this hack :)
Outerbounds | Systems dev | SF / Remote | Full-time
Outerbounds was founded recently to commercialize Metaflow, an open-source Python framework that makes infrastructure easily accessible for machine learning/data science projects. Metaflow was originally started by us at Netflix and it is now used by hundreds of companies across industries.
We care a lot about thoughtful design, overall product experience, and quality of code. We are looking for backend/systems engineers who are experienced in at least one of the following areas: Delightful API design, deep Python experience, distributed systems, or low-level systems programming.
- Metaflow: https://docs.metaflow.org
- More about us here: https://outerbounds.com/workwithus
- Join our Slack to see the project in action: http://slack.outerbounds.co
- Email: workwithus@outerbounds.co
Yep, Kedro and Metaflow are more similar to each other than to other generic DAG orchestrators like Airflow.
Kedro and Metaflow make it easier to develop robust ML projects where orchestration plays an important role but it is not everything. They are two separate projects, so the way how they approach the problem differs greatly in details.
Outerbounds | Systems dev | SF / Remote | Full-time
Outerbounds was founded recently to commercialize Metaflow, an open-source Python framework that makes infrastructure easily accessible for machine learning/data science projects. Metaflow was originally started by us at Netflix and it is now used by hundreds of companies across industries.
We care a lot about thoughtful design, overall product experience, and quality of code. We are looking for backend/systems engineers who are experienced in at least one of the following areas: Delightful API design, deep Python experience, distributed systems, or low-level systems programming.
- Metaflow: https://docs.metaflow.org
- More about us here: https://outerbounds.co/workwithus
- Join our Slack to see the project in action: http://slack.outerbounds.co
- Email: workwithus@outerbounds.co
Outerbounds | Systems dev | SF / Remote | Full-time
Outerbounds was founded recently to commercialize Metaflow, an open-source Python framework that makes infrastructure easily accessible for machine learning/data science projects. Metaflow was originally started by us at Netflix and it is now used by hundreds of companies across industries.
We care a lot about thoughtful design, overall product experience, and quality of code. We are looking for backend/systems engineers who are experienced in at least one of the following areas: Delightful API design, deep Python experience, distributed systems, or low-level systems programming.
- Metaflow: https://docs.metaflow.org
- More about us here: https://outerbounds.co/workwithus
- Join our Slack to see the project in action: http://slack.outerbounds.co
- Email: workwithus@outerbounds.co
There are a few different ways to deal with spot interruptions. First, it is a good idea to specify multiple instance types in your compute environment so even if some instances types become unavailable in spot, Batch can use another type automatically.
Second, you can rely on Spot Fleets which handle both spot and on-demand instances seamlessly https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/spot-fle...
Outerbounds | Frontend dev | SF / Remote | Full-time
Outerbounds was founded recently to commercialize Metaflow, an open-source Python framework that makes infrastructure easily accessible for machine learning/data science projects. Metaflow was originally started by us at Netflix and it is now used by hundreds of companies across industries.
We care a lot about thoughtful design, overall product experience, and quality of code. We are looking for stunning colleagues who can help us build delightfully usable, technically non-trivial UIs. This is a great greenfield opportunity for a senior frontend engineer who wants to build a product from scratch.
- Metaflow: https://docs.metaflow.org
- More about us here: https://outerbounds.co/workwithus
- Join our Slack to see the project in action: http://slack.outerbounds.co
- Email me: ville@outerbounds.co
As the article mentions, Metaflow will start supporting Kubernetes natively soon, although data scientists don't need to care about it :) Nothing changes in your Metaflow code when you move e.g. from AWS to Azure, so Metaflow isn't fundamentally dependent on AWS in any way.
Netflix is an AWS shop, so naturally we started with AWS integrations.