New Apache Airflow Operators for Google Generative AI

https://cloud.google.com/blog/products/data-analytics/announcing-apache-airflow-operators-for-google-generative-ai

by seeyam • 2 years ago

42 29 2 years ago

cloud.google.com

ssahoo2• 2 years ago

So they added three brand new Airflow operators to interact with Vertex AI's generative models to their Cloud Composer

I can't imagine why people use Airflow these days. Its DAG DSL means you have to fit your biz logic to their paradigm. I expect a framework to fit nicely with existing biz logic. And that means you're essentially stuck. It has no interoperability; hence the need to build custom operators for everything.

It doesn't scale for teams because the package is incredibly bloated, so once you need to run multiple images via K8s operators, you lose out on a lot of other Airflow functionality because it all assumes you have your biz logic embedded within DAGs, but at the same time I don't know why anyone would develop this way.

ssahoo2• 2 years ago

What alternative do you recommend?

annexrichmond8• 2 years ago

I've done a lot evaluation of such frameworks, and I hope to publish more on it. It really depends on your requirements. I would look at Prefect, Flyte, Dagster, and Temporal ahead of Airflow, though.

computershit• 2 years ago

Prefect has more polish and is easier to get started than any of the existing options. We've been running their self-hosted for over three years and it basically stays out of the way.

We looked at Dagster as well as Airflow. I really, really liked Dagster but the BI team didn't.

I cannot imagine using Airflow for anything meaningful and respecting myself at the end of a work day. The local development experience was abysmal. Deployments sucked.

That being said, if you're not using anything except maybe cron right now, and if you don't care about the solution being a proper data pipelines orchestration platform trademark symbol, I'd recommend starting with Windmill.

liveoneggs• 2 years ago

jenkins

nyrikki3• 2 years ago

The DAG constraints are a feature to prevent architectural erosion.

Note that any feedforward neural network (e.g. anything using attention) is also a DAG.

You can always encapsulate more complex logic in a single subroutine or even choose a hex or onion pattern for biz logic.

But what do you suggest besides DAGs + saga patterns that doesn't result in a ball of mud over time for distributed systems?

annexrichmond8• 2 years ago

DAGs are fine. Their DSL is not because it's abstracting the wrong things in the wrong place. It's a global file with static definitions; why do you hardcode KubernetesOperator when maybe you don't want a KubernetesOperator in a test env? There is also no type safety between tasks/operators. And it's an extremely dependency heavy package with no client/server isolation so bundling Airflow for multiple teams is just not viable.

lyu072822• 2 years ago

hence the need to build custom operators for everything

But isn't that the point? I never got the impression that you were supposed to build everything with the built-in operators, they are just the "batteries included" part you can wrap or extend. I really don't understand the criticism.

My criticism was always xcom, but that's a moot point now that we have TaskFlow. Airflow is awesome and very flexible you just have to adapt to how it's supposed to be used instead of fighting against it I find.

annexrichmond8• 2 years ago

That might be fine for small teams, but any 100+ person company would already have their own abstractions and don't need to reinvent those wheels just for Airflow. But even if you're smaller I still think you're setting up for failure if all your biz logic is within Airflow unless you are careful about making it reusable/shareable for other contexts. Eg, what if you want to convert some scheduled pipeline to some event-driven architecture with other systems? That means needing to refactor everything out. It's not interoperable or modular and that's why it should be avoided.

politelemon• 2 years ago

I expect a framework to fit nicely with existing biz logic.

I wouldn't, that's just describing a custom codebase. Airflow comes with bells and whistles that can be taken advantage of if you fit your execution in their DAG model, that's all. Those bells and whistles can be an excellent pattern to work with. You can go all in on operators, or you can just have an operator call out to an external task that does all the work and only rely on Airflow for its retry/alerting mechanism while keeping the external task in language of your choice.

annexrichmond8• 2 years ago

Airflow owning the scheduling, retry, task branching/dependencies is fine. But a few issues: to take advantage of Airflow's core features (XCOMs, task mapping, etc) your biz logic/tasks must be in the same venv, which is not sustainable as teams grow. Once you have external tasks, you now have a more complex system to operate and test, and your Airflow installation is now very overkill (you need a pool of workers just to monitor external... workers?).

manishsharan• 2 years ago

> I can't imagine why people use Airflow these days.

GCP Dataflow is based on Airflow. I imagine a lot of GCP customers would default to this for their data workflow

KptMarchewa• 2 years ago

No, it's based on Apache Beam. Cloud Composer is the GCP's Airflow distribution.

otabdeveloper4• 2 years ago

People use it because it comes with a fully functional web UI. (That's really the only reason why it's used. If it was just ad-hoc bash scripts underneath the UI I don't think anybody would mind.)

braza• 2 years ago

I can't imagine why people use Airflow these days. You lose out on a lot of other Airflow functionality because it all assumes you have your business logic embedded within DAGs.

As an old Airflow user, I can relate with that.

The contributors are great and the speed of the new features and fixes are great also.

I see two main issues with Airflow: (i) the lack of good interoperability with a cloud native stack, specifically with the k8s operators, that sounds quite hackish and (ii) lack of a better version control in the DAGs in way that we can have actually real lineage.

The point is that for most of the veterans in ~ETL~ Data Engineering in Cloud Native stacks, we just want to send a cli command or a Make command to a container to execute the thing that we want.

Apache Airflow does that, but as most of the things in the Python ecosystem at the time that you have a great UI (not confuse with UX) become way easier to embed business logic in the DAGs and have your application embedded with Airflow logic.

I will never bash Airflow since I got a lot of money in consulting migrating from and to it, but the general feeling of working on a daily basis for almost 7 years made me realize the sad reality around orquestrators. In the past, those used to be kind of the bedrock technology that you would never want to migrate, but today, due to this feature bloat and lack of vision, I started to see orquestrators as disposable as Java Script frameworks.

New Apache Airflow Operators for Google Generative AI

Related Stories