Fast, Ordered Unixy Queues over NNCP and Syncthing with Filespooler

https://changelog.complete.org/archives/10368-fast-ordered-unixy-queues-over-nncp-and-syncthing-with-filespooler

by pabs3 • 4 years ago

34 2 4 years ago

changelog.complete.org

tarasglek• 4 years ago

I didn't take a deep look at this, but the general idea of setting up a worker queue as a filesystem structure & synchronizing via sync-thing is pretty cool.

Only question is, would this support multiple-workers? Would be neat to be able to distribute work a to a cluster this way.

jgoerzen• 4 years ago

I think Syncthing is an under-appreciated tool! I wrote more about it at https://changelog.complete.org/archives/10219-a-simple-delay...

https://www.complete.org/parallel-processing-of-filespooler-... discusses parallel processing in Filespooler. This isn't a design goal and may not be the best fit, but when everything's just a file, some tricks are possible here.

Another interesting use case is one-to-many and many-to-many; https://www.complete.org/filespooler/ describes those.

Fast, Ordered Unixy Queues over NNCP and Syncthing with Filespooler

Related Stories