Recurring Jobs

Sometimes you may need to run a job regularly. For example, you must run an analysis program on the data collected daily. This means you need to set a reminder to repeat the same at the end of every day. Slurm allows users to schedule recurring jobs through scrontab.

Scrontab File Structure

scrontab file can contain comments starting with '#', whitespaces, valid crontab entries or custom variable settings. Lines that start with #SCRON are treated like the beginning of a new batch job, and work like #SBATCH directives for batch jobs. They allow options to be defined for the single following crontab entry.

scrontab command is the main entry of the file. It defines the interval to run the sheduled reccuring job. The format of the scrontab command looks like the following. m h dom mon dof command, where,

  • m represents the minute [0-59].

  • h represents the hour [0-23].

  • dom represents the day of month [1-31].

  • mon represents the month [1-12] (or name).

  • dof represents day of week [0-7] (0 and 7 are Sunday, or use name).

  • command represents the reccuring job.

For example, 5 20 * * * command1 runs command1 everyday at 8:05pm. Note the asterisk * which means that it’s valid for each of the allowed values for the given time period. To run the same commad every Monday at midnight you may type, 0 0 * * 1 command1. Note that the value for minute and hour is 0 to make sure that the command runs once on Monday.

If you have a command that you need to run every Monday and Friday, you can use a comma operator to specify multiple values as shown bellow.

0 0 * * 1,6 command1

To run a job at every specfic interval, you can use the forward slash operator /. For example to run command1 every three days, type:

0 0 */3 * * command1

scrontab allows you to use shortcuts to specify some common time intervals for the specified script to run. These include the following:

  1. @yearly | @annually: job will run at 00:00 Jan 01 each year.

  2. @monthly: job will run at 00:00 on the first day of each month.

  3. @weekly: job will run at 00:00 Sunday of each week.

  4. @daily | @midnight: job will run at 00:00 each day.

  5. @hourly: job will run at the first minute of each hour.

Slurm cannot guarantee that the jobs will run at the requested times, but tries to schedule it as close as possible. Internally, jobs are submitted to Slurm and set with a specific BeginTime. Once a scron job ran, it will be requeued with a new BeginTime. These jobs will also show up in squeue.

Scrontab Command

scrontab is used to set, edit, and remove a user’s Slurm-managed crontab file. This file can define several recurring jobs to run jobs at a specific interval.

Syntax

scrontab [-u user] [ -e | -l | -r ]

Where,

  • -u user to edit crontab for.

  • -e edits user’s crontab. If a crontab does not exist already, a default example (without any defined entries) will be provided in the editor. The currently running reccuring jobs continue, but no more reccuring jobs later.

  • -l lists user’s crontab.

  • -r deletes user’s crontab.

To edit your scrontab file run:

 scrontab -e

If a scrontab does not exist already, a default example (without any defined entries) will be provided in the editor.

To list all the current scrontabs, run:

scrontab -l

At anytime you can delete your scrontabs by running:

scronab -r

Examples

Example1

To print the date every day at 8 am, you must edit the scrontab using the command scrontab -e and add the following to it:

#SCRON -p normal
#SCRON --chdir /path/to_output
#SCRON -o %j-out.txt
0 8 * * * echo $(date +%F)

-p defines the partition to run the job. --chdir to define the output path, because the output is placed in the home directory by default. -o names the output file.

Example2

Suppose you have a job script example_job.sh with the following content:

#!/bin/bash

#SBATCH --job-name=TestJob
#SBATCH --ntasks=3
#SBATCH --time=00:05:00
#SBATCH --cpus-per-task=2
#SBATCH --mem-per-cpu=1G   # memory per CPU core
#SBATCH --partition=normal  ## the partitions to run in (comma seperated)

srun --ntasks=1 --nodes=1 --cpus-per-task=$SLURM_CPUS_PER_TASK bash -c "sleep 10; echo 'hello 1'" &
srun --ntasks=1 --nodes=1 --cpus-per-task=$SLURM_CPUS_PER_TASK bash -c "sleep 20; echo 'hello 2'" &
srun --ntasks=1 --nodes=1 --cpus-per-task=$SLURM_CPUS_PER_TASK bash -c "sleep 30; echo 'hello 3'" &

And you need to run this job monthly, edit the scrontab and add the following:

INPUTDIR=/path/to_the_job
#SCRON --chdir /path/to_output
@monthly $INPUTDIR/example_job.sh