Show HN: Dcron – Distributed, fault tolerant job scheduling system

http://dcron.io/

by victorcoder • 11 years ago

42 24 11 years ago

dcron.io

dozzie2• 11 years ago

You may want to rename your project. dcron is Dillon's cron for quite a lot time already.

feld• 11 years ago

Yeah upon reading the title I thought Dillon had done some improvements on dcron.

jedisct1• 11 years ago

Yup, thought the same thing.

victorcoder6• 11 years ago

Right I know, but if he don't mind, both projects are enough different to not create confusion.

Need more time to see if this will be necessary.

dozzie2• 11 years ago

No, they are not different enough. Both are cron daemons. Both are in the end supposed to be packaged. It's really enough to make the newer one (newer by decades!) to change its name.

mattzito• 11 years ago

Honestly, it's just the classy thing to do - the impact of changing your name is nominal, the potential confusion with the other Dcron is significant.

victorcoder6• 11 years ago

The project has been renamed to Dkron due to respect to Dillon's cron and to avoid confusion.

nodesocket2• 11 years ago

How do I define jobs? Can they be written in any language? I didn't see anything in the documentation.

kylequest• 11 years ago

Looks like it's defined using the format you see in this json file: https://github.com/victorcoder/dcron/blob/master/job.json

The "command" field is the most important part. What you put there gets executed like this by their agent (on Linux): /bin/sh -c <your_scheduled_command_name>

andyeskridge• 11 years ago

I was looking into that too. Based on this job.json[0] definition from the git repo, it looks like it just executes the command. So I believe you could have the job written in any language.

But I think this also means you have to copy every job executable to every server that might have a chance of running the job.

[0] https://github.com/victorcoder/dcron/blob/master/job.json

victorcoder6• 11 years ago

Added to the docs, big omission, sorry! http://dcron.io/docs/api/#create-or-update-a-job

nodesocket2• 11 years ago

Any change that dcron could manage the job scripts itself? I.E. you store the raw command scripts in dcon, then reference them in the create a job endpoint?

For example:

    POST /scripts --data 'node /home/node/myJob.js'

That returns a script id { "script_id": "507f1f77bcf86cd799439011" }

Then:

    POST /jobs --data '{
       "name":"cron_job",
       "schedule":"@every 2s",
       "script": "507f1f77bcf86cd799439011",
       "owner":"foo",
       "owner_email":"foo@bar.com"
    }'

This way you don't have to manage/create the scripts on each node manually.

Abundnce10• 11 years ago

I have a bunch of Python scripts that run a various intervals (hourly, daily, monthly, etc). I have trouble keeping track of which jobs fail -- unless I notice an email doesn't show up, or a metric is missing. Is Dcron something I would use to keep track of my cron jobs and whether or not they ran successfully?

At this point I don't need my jobs to be distributed, I'm just looking for a way to keep track of them all and visualize characteristics of the job (start time, length, successful, etc). Ideally, I'd have a way to re-run the job within the UI.

yashap• 11 years ago

Check out Airflow (open source project from airbnb that does all of that very well).

victorcoder6• 11 years ago

Sure, it can do this well. It can run in a single dcron/etcd node if you don't need it to be fault tolerant. At this point you should be checking the job status in the ui as it doesn't have notifications yet.

abrookewood• 11 years ago

You can also try cronitor and others like it. You add a CURL command at the end of your script that 'touches' a HTTPS endpoint. If your script doesn't check-in at pre-defined intervals, you get alerted. They are perfect for situations where you don't actually want to set up any infrastructure.

fortyfivan• 11 years ago

Disclosure: I work for Iron.io

If you're open to a hosted solution, check out IronWorker: http://www.iron.io/worker

It's an async task processing service with a built-in job scheduler. You can upload your python scripts to Iron.io, then set schedules and other triggers to execute on-demand. We have a dashboard to manage tasks and schedules, see what ran and what failed, and you can visualize the characteristics you're looking for. We do distribute the workloads for you, but sounds like it could be a good fit.

JonM• 11 years ago

Hey,

I've been tinkering with this concept recently, it's not quite ready for beta testing, but I'd be happy to take feedback on what I've got so far: http://croncloud.io/

carl_• 11 years ago

Cool! Might use this when I renovate http.sh

nemothekid• 11 years ago

I haven't done any digging, but I'm assuming this uses etcd for membership? How does that work - how does a server know whether its a leader, and how does failover work?

victorcoder6• 11 years ago

Nope, it uses serf for membership.

endymi0n• 11 years ago

Man, how I wish Kala and Dcron would just work together.

https://github.com/ajvb/kala

https://github.com/victorcoder/dcron

Fusing both would be AWESOME (I'd even like to help).

What Dcron gets right:

- Distributed by design

- Raft for the reliable parts, gossip for lightweight sync, a match made in heaven

- Linux mentality

- Dashboard included

- Good docs, nice website, looks serious

What Kala gets right:

- Built much more directly after Chronos, which has an amazing and battle tested API

- Using ISO durations instead of NIHing their own format

- Dependent jobs

schmichael• 11 years ago

While Metafora doesn't support cron-like syntax for job scheduling or tag-based node targeting (yet!), it does support distributed fault tolerant task running. We hope to build out a webui soon as well.

However since Metafora is written as a Go library and provides a statemachine implementation for task handlers to use, it requires writing your distributed tasks in Go.

https://github.com/lytics/metafora

Show HN: Dcron – Distributed, fault tolerant job scheduling system

Related Stories