Ask HN: How do you monitor your background jobs?

https://news.ycombinator.com/item?id=22712908

by kostarelo • 6 years ago

1 1 6 years ago

news.ycombinator.com

When you have many background jobs, that take either a few seconds or lots of minutes/hours, how do you monitor them?

How do you know when they started and when they finished, their outcome, etc. Do others, non-tenchical folks, need access to that or is it just the developers?

simonblack• 6 years ago

Most background jobs are run by cron.

Cron has the capability to set environment variables. The scripts/programs can check those variables and then have the ability to adjust the type and quantity of logging according to whether they are run as cron jobs or from the command-line.

The logging of these jobs goes to a specific log file which is rotated daily. The daily logs are kept for 10 days before being discarded.

For instance this is a cron entry. Note the logfile argument as $1.

     # update BAK_MONTHLY on 2nd of the month only
     24 09 2 * *         /home/jvs/binscripts/rsync.bak.monthly.xpz /var/log/cronlog_jvs

This is cron setting variables in the crontab file

     # DO NOT EDIT THIS FILE - edit the master and reinstall.
     # tell cronjobs that they are running under cron
     CRONJOB=CRONJOB
     # specify the logfile for cronjobs output
     LOGFILE=/var/log/cronlog_jvs
     # expand the path
     PATH=/usr/sbin:/sbin:/usr/local/sbin:/usr/bin:/bin:/usr/local/bin:/home/jvs/binscripts
     #
     # m h  dom mon dow   command

and this is a cronjob template

     #!/bin/bash
     #jvs script
     #
     # DATE WRITTEN ......
     #
     # This crontab item does ........
     #
 
     #Set variables
     TARGET=
     NOW=$(date +%y%m%d)
     DAY=$(date +%d)
     LOGFILE="/tmp/${HOSTNAME}.crontab-item.log"
 
     if [ "$1" != "" ]; then
        LOGFILE="$1"
     fi
 
     touch "$LOGFILE"
     chmod 666 "$LOGFILE"
 
     echo -e "#######################################################\n " >>"$LOGFILE"
     /bin/date +"%F %H%M" >>"$LOGFILE" 2>&1
     echo "$0" >>"$LOGFILE"
 
 
     ### ***************************************************
     ### *******   Guts of crontab item goes here   ********
     ### ***************************************************
 
     # Say when this item completed
     echo -e "\n$0 Completed at $(date)" >>"$LOGFILE"

Ask HN: How do you monitor your background jobs?

Related Stories