asksol

I'm not sure operators are a good example, they can be idempotent or not depending on the implementation. The operator is a function that takes two values and return a result: result = op(left, right).

But that safety net doesn't extend to startup founders does it? AFAIR In Norway, you need to have worked in a company as a regular employee for at least one or two years before you can claim unemployment benefits. Health insurance is always there, and so is basic social benefits (but the unemployment one is the only you can live on).

In Faust we've had the ability to take a very different approach towards quality control. We have a cloud integration testing setup, that actually runs a series of Faust apps in production. We do chaos testing: randomly terminating workers, randomly blocking network to Kafka, and much more. All the while monitoring the health of the apps, and the consistency of the results that they produce.

If you have a Faust app that depends on a particular feature we strongly suggest you submit it as an integration test for us to run.

Hopefully some day Celery will be able to take the same approach, but running cloud servers cost money that the project does not have.

Yes, they do serve different purposes but they also share similarities. You could easily write a task queue on top of Faust.

It's important to remember that users had difficulty understanding the concepts behind Celery as well, perhaps it's more approachable now that you're used to it.

Using an asynchronous iterator for processing events enables us to maintain state. It's no longer just a callback that handles a single event, you could do things like "read 10 events at a time", or "wait for events on two separate topics and join them".

Faust is a library that you can import into your Python program, and all it requires is Kafka. Most other stream processing systems require additional infrastructure. Kafka Streams has similar goals, but Faust additionally enables you to use Python libraries and perform async I/O operations while processing the stream.

We had support for it initially, but we ended up not using the schema registry server. We want to add support for this back in if it provides value.

There's an examples/django project in the distribution. I think they removed the gevent bridge from PyPI for some reason, but you can still use the eventlet one. Gevent is production quality and the concept of bridging them is sound, so hope someone will work on it.

We added two levels of abstractions for this purpose:

- A Stream iterates over a channel

- A Channel implements: `channel.send` and `channel.__aiter__`.

- A topic is a "named" channel backed by a Kafka topic

- Further the topic is backed by a Transport

- Transport is very Kafka specific

To implement support for AMQP the idea is you only need to implement a custom channel.

If you open an issue we can consider how to best implement it.

Faust can serve the data over HTTP/websockets and other transports, so you can query it directly!

I'm looking at Redis Streams and it seems to lack the partitioning component of Kafka. One of the properties that we rely on is the ability to manually assign partitions such that a worker assigned partition 0 of one topic, will also be assigned partition 0 of other topics that it consumes from.

Simplicity is of course a goal, but this may mean we have to sacrifice some features when Redis Streams is used as a backend.

I'm glad you like Celery, this project in many ways realize what I wanted it to be.

My name is Ask and I am one of the co-creators along with Vineet.

Thanks for pointing this out. Fixed the links. You can also find the docs here: http://faust.readthedocs.io/en/latest/

Faust uses Kafka for message passing. The new messages you create can be pushed to a new topic and you could have another agent consuming from this new topic. Check out the word count example here: https://github.com/robinhood/faust/blob/9fc9af9f213b75159a54...

Also note that the Table is persisted in a log compacted Kafka topic. This means, we are able to recover the state of the table in the case of a failure. However, you can always write to any other datastore while processing a stream within an agent. We do have some services that process streams and storing state in Redis and Postgres.

My name is Ask and I'm the co-creator of this project, along with Vineet.

Yes, we're definitely interested in supporting Redis Streams!

Faust is designed to support different brokers, but Kafka is the only implementation as of now.

My name is Ask and I'm co-creator on this project, along with Vineet Goel.

This answer sums up why we wanted to use Python for this project: it's the most popular language for data science and people can learn it quickly.

Performance is not really a problem either, with Python 3 and asyncio we can process tens of thousands of events/s. I have seen 50k events/s, and there are still many optimizations that can be made.

That said, Faust is not just for data science. We use it to write backend services that serve WebSockets and HTTP from the same Faust worker instances that process the stream.

You don't think Equifax have lawyers? Of course they do, and they know that the clause exists. Just having the clause in the first place is a conspiracy to abuse consumers.

Celery was never meant as a replacement for cron, it was simply a nice bonus that fits the messaging pattern well. Writing a task queue is actually very simple using for example Redis, but that doesn't necessarily mean Celery is over-engineered IMHO. It's very easy to forget the support required once your system is in production.

Disclaimer: I'm a contributor

Thank you for the kind words, it's so very appreciated :)

I have merged many features, like broker transports, result backends, etc, and while the initial contribution was great, it ends up being unmaintained with issues that nobody fixes.

If there's any feature that you really want back, chances are the problems with that feature are not super difficult to fix, so please reach out!

Definitely questionable, but maybe it's related to germs transferred during birth: http://well.blogs.nytimes.com/2016/02/01/post-cesarean-bacte...

I use hjkl consistently in vim and in the shell. Just think about how long it takes you to move your fingers to the cursor keys. You should spend some effort getting used to it, as using hjkl means you can keep your hands in the middle of the keyboard. I have also never had any problems with strain, and I suspect this is part of the reason.

I agree it has some advantages, but citing commonality seems weird when it's increasingly unlikely for new to be homeowners to afford a home in cities like London, SF, etc. I doubt most people waste thousands of dollars each month for the ability to relocate quickly?

Does it really apply to just browser histories? Does it mean they can take your computers without warrant to get to this history, or does it mean ISPs will have to sniff traffic to extract URLs visited and keep a log of them (for how long?).

It doesn't make much sense to me, unless they are carefully wording this into something that seems reasonable to the public ("I'll just use private mode, no deal") when it really means monitoring all our internet traffic. But then why is the media repeating it?

Edit: Ignorance is bliss

I found your project very interesting! I'm very familiar with Erlang/OTP, and have been meaning to play with Elixir for some time. Since you're separating downloading from processing, maybe IPC is a better term for what you're doing, as you don't want these two steps to be separated over the network for data locality.

I only wanted to clarify the RAM situation described in your slides, as it's not widely known that you can use eventlet/gevent with Celery.

Author of Celery here. This is an interesting presentation!

I want to clarify something about Celery and RAM usage.

When writing web crawlers, and other (mostly) I/O-bound tasks, you should be using the eventlet/gevent execution pools instead of the multiprocessing one. This will drastically reduce memory use, and perform better.

If you have four CPU cores you can start four worker instances with 1000 threads each (for a total of 4k threads): `celery multi start 4 -A proj -P gevent -c 1000`

This will utilize all the CPU/cores in your system, working around the GIL.

One of the new features coming in Celery 4 is a message protocol with support for multiple languages, maybe we could have an Elixir worker soon.

People need more venues to meet new people, but by greeting I meant more like saying "evening!" when you pass your next door neighbor in the hallway. I don't think there's an expectation to chat in this case. If I wanted to chat I'd start with "excuse me," and probably walk up first as you say.

A greeting can be made from across the street, and sometimes just lifting your hat in response is enough (I don't have one, but I have seen people do this).

I was also brought up to greet people, but try to greet neighbors in London, UK and they will look at you like you fell from the sky. Most of them will not even give a response. Big cities are a lot like this now.

I think it's more readable in isolation, but when you have workflows it distracts you from what it's actually doing, making it harder to see what the purpose is at a glance. An API can probably not have many shortcuts like this, but you learn what .s does once, and then you barely notice it.

But that's my opinion and I have come to learn not everybody value succinctness in code, so you have the choice of both :o)

Had I not been constrained by backward compatibility I may have made it so that task(arg) only defines the signature of a task invocation and you'd need to do task(arg).delay() to call it remotely, and task(args)() to call it as a function locally.

Be careful using the multiprocessing module, it has some very serious bugs. I've spent the last 4 years rewriting parts of it for Celery

I regretted this as soon as I submitted it. I would hate for someone to do the same thing to my projects so I should know better. I've written about it before, but realize that you probably have not read it :)

I really like the multiprocessing library, it helped me start Celery in the first place. What it tries to solve is actually very very complicated, and you would have to test it on production systems for years to be sure it works, and I think Celery was the app that did that testing. I contributed some fixes back into Python, but most of it is not merged upstream.

The most complicated issue I had to solve was that multiprocessing.Pool uses POSIX semaphores to share pipes between processes (that's how the pool processes receive jobs, and the parent receive results). If a child process is killed before releasing that semaphore you have a deadlock that's tricky, if not impossible to solve. So I rewrote the pool to use async I/O instead, which also had the side effect of drastically improving performance (no locks). Sadly I'm not sure how to implement that on Windows, so it's unlikely to be merged upstream. Other fixes and features used by Celery is available in our billiard (on PyPI) fork of multiprocessing, but the async pool is not part of that yet as it currently depends on code in celery that does not fit in billiard (it should be rewritten to use asyncio now).

You can claim to replace Celery using a small layer on top of async I/O, or claim to replace Celery with a simple Redis list operation, but I think that's unfair to all the work that went into Celery, and the other features Celery implements like monitoring, workflows, and a large list of other things that you don't immediately think of when starting a project. It keeps a repository of these patterns for the Python community, and even something like crossbar.io could be supported as a transport.

This is a very limited type of communication. Asyncio with stuff like crossbar.io allow pub/sub.

Celery also supports pub/sub, and other topologies.

So can your celery workers. It happened to me many times.

With the major difference that your tasks can be redelivered to a different worker, and so will complete anyway.

Well, the main point of the article is that celery is solving >the GIL. It's not, it's bypassing it

I was agreeing with you there, but I guess my reply was not clear on that. I just wanted to point out some inaccuracies in your reply.

coroutines and multiprocessing

Be careful using the multiprocessing module, it has some very serious bugs. I've spent the last 4 years rewriting parts of it for Celery

Celery can use async I/O too! Using the gevent pool you can have e.g. 4 celery worker instances with 1000 threads each.

- You are limited to x number of blocking operations, where x >is the number of workers. It means you cannot do massively >parallel I/O such as network operations with it (e.g: a web >server).

Celery can use eventlet/gevent instead of multiprocessing for executing tasks, so this should be possible (granted, not sure if using it as a web server is a great idea)

- Tasks cannot communicate with each others;

This is not true, they can send messages to each other

- You must juggle with the workflow of your tasks (is it ready >? it it dead ?). You can use await stuff() with a try/except;

If you have to juggle it means your workflow design is not good enough. I.e. you shouldn't wait for other tasks, you should have callbacks and errbacks.

Also, note that a try/except does not guarantee the operation will be completed (e.g your asyncio app can be killed).

I'm not sure there's much point in comparing these, they are wildly different concepts: Celery is a distributed system, asyncio is for async I/O, the GIL is only a problem if you can't start n instances of your app.