Plan – Cron jobs in Python

by fsp • 12 years ago

57 18 12 years ago

chhantyal• 12 years ago

And if you are already using Celery, it has crontab feature http://celery.readthedocs.org/en/latest/userguide/periodic-t...

jocmeh• 12 years ago

Yeah, we are using that as well. It's nice because it hooks up to your Django admin and people with the right permissions can reschedule stuff.

I don't see any use case for Plan, it's just complicating an easy task.

gtuckerkellogg• 12 years ago

I can see one use case for this family of tools: if the same script could generate either crontab files or launchd agents, so it could generate scheduling files for MacOSX, where crontab is discouraged.

crdoconnor• 12 years ago

Celery beat is pretty much exactly the same thing as cron except it's another service to configure and worry about keeping up and running. And it uses 35 MB of RAM instead of...well, I guess cron uses a few kilobytes.

I can't see the point of it at all.

When I want to kick off celery tasks I just have a cron job execute a django shell command.

icebraining2• 12 years ago

I guess cron uses a few kilobytes.

A full megabyte on my machine, actually! Still a long way from 35MBs, though.

Walkman3• 12 years ago

I don't get it. What's so hard about cron jobs that you have to write a wrapper for it? It is easy to understand if you just read the crontab file comment at the top. There is just two commands:

    crontab -l

for listing

    crontab -e

for editing. What else do you need?

With this, instead of learning crontab simple syntax, you need to:

- install this package

- learn its API

- run it and still edit the crontab file :D

seems like not worth it.

ddv_• 12 years ago

Whenever gem does the same thing. So you can keep it under version control with your project.

icebraining2• 12 years ago

It's just a text file, why not just add it to the version control of your project?

If it's because users can only have a single crontab, you can always use system crontabs instead or "cat" them together.

Walkman3• 12 years ago

You can install a new crontab file with

    $ crontab [ -u user ] file

est• 12 years ago

How about dynamic periodic tasks?

For instance, server side exponential backoff.

noselasd• 12 years ago

This approach has a human in the loop. There's plenty of use cases where you want a program to edit cron jobs.

asimjalis• 12 years ago

Here is some brainstorming on use cases.

1. You can validate cron jobs and programmatically change or reject some of them. Or maybe just offer a small set of services as a cron job.

2. Once you have a security model you can expose this as a web service and offer cron jobs over the web.

3. You could extend this API to use different implementations of cron. Instead of calling the actual cron you could implement cron in Python or maybe use a cron web service.

_navaneethan• 12 years ago

This would be useful incase, if you want to set cron dynamically.Based on the executed result, next step can be manipulated via this.

Walkman3• 12 years ago

1. crontab -e command automatically recognize if you have a syntax problem in the newly edited file:

    $ crontab -e
    crontab: installing new crontab
    "/tmp/crontab.QhkOLb/crontab":25: bad minute
    errors in crontab file, can't install.
    Do you want to retry the same edit? (y/n)

2. See better solutions (like Celery) below.

nlindblad_se• 12 years ago

I wrote something similar a while back, but it only requires an hourly cronjob to run a bunch of Python scripts in an arbitrary file structure: https://github.com/nlindblad/ocarina

I use it extensively myself for scraping various sites/APIs.

talideon• 12 years ago

I'd typically deal with this by exporting the functions to be ran as entry points in the the setup.py file, and then use whatever configuration management tool I'm using (Ansible, Puppet, whatever) to manage the crontab file in /etc/cron.d. Ansible's cron module makes this especially easy.

IgorPartola• 12 years ago

I see the value in this: it provides a nice programmable interface to cron. Unlike what you might think at first, cron has some really interesting idiosyncrasies that you might not appreciate at first. For example, if your filename in /etc/cron.d/ has a "." character in it, it will be silently skipped [1]. Also, files in /etc/cron.d/ will not inherit environmental variables from /etc/crontab. Consider how many hours have been wasted "debugging" these issues.

That said, cron is easy to abuse as well. The example of providing exponential backoff is kind of silly: there are better tools for the job, including Celery. Cron also does not have seconds precision which is a bummer for lots of short-lived jobs. Lastly, cron only runs jobs at the specified time. If you adjust the system clock, or shut the system down, your tasks might be skipped. anacron [2] might be a better solution in some ways. Lastly, all implementations of cron that I am aware of don't have a great ability to make sure tasks actually run and finish, don't have retries built-in, and have a horrible interface for reporting errors (any output of the script is emailed to root@localhost from root@localhost).

Based on the above, you might think I am saying to never use cron. That's not at all true. Cron is extremely useful as a systems tool. If you package your software in the distro's package format (you should at least give this a try to see how the system is put together), cron is pretty much your only safe choice. Also, as long as you follow the rules cron sets up, and know of its potential issues, cron will be the workhorse that gets things done. Tasks like log rotation, periodic sanity checks, etc. are great for this. If you are writing a web application and are already using Python though, look at more sophisticated options such as Celery before using cron directly.

[1] http://manpages.ubuntu.com/manpages/maverick/en/man8/cron.8.... and https://bugs.launchpad.net/ubuntu/+source/cron/+bug/706565

[2] http://en.wikipedia.org/wiki/Anacron

Plan – Cron jobs in Python

Related Stories