Job Composer

To create jobs through Open OnDemand, navigate to Jobs > Jobs Composer from the dashboard.

Jobs Composer

You will see the below output opened in a new tab. The list of options available to create new jobs are point_right From Default Template, From Template, From Specified Path, and From Selected Job.

Jobs Dashboard

Create Job

From Default Template

  1. On the Job Composer page, select New Job > From Default Template.

    Open Jobs Template

    A new job will be created like below.

    Create Job
  2. You can modify the options of the created job like name, Cluster, Job Script using the Job Options button.

    Modify Job Options

    After you click the button, you can change the job options like below.

    Modify Job Options

    In the above example, the Name has been changed from default (Simple Sequential Job) to TestJob. AFter the name has been modified, hit the Save button and the modified job name will be replicated in the Job Composer page like below.

    Output:

    Modify Job Options
  3. Now, modify the submit script main_job.sh.

    1. On the Job Composer page, hit the Open Editor button under Submit Script section.

      Modify Job Script
    2. In the text editor opening in a new tab, modify the Job Script with the below content.

      Modify Job Script
    3. After the job submission script main_job.sh has been updated with the above code, hit the Save button. Now, go back to the previous tab and you will see the submission script updated.

      Modify Job Script

      The above Job Details pane also shows important information related to the job.

      S.No

      Value

      Explanation

      1

      Submit to: discovery

      Name of the cluster that job will be submitted to.

      2

      Script Location: home/<username>/ondemand/data/sys/myjobs/project/default/1

      Path to the submission script location.

      3

      Script Name: main_job.sh

      Name of the submission script file.

      4

      Folder Contents: main_job.sh

      Folder contents of the submit script file.

      If you want to open the submit script main_job.sh file directly in the terminal, use the >_Open Terminal button. Open Dir button will open the directory of the submit script in file manager.

  4. Now, hit the submit button to submit your Test Job. Thus, the status will be changed from Not Submitted to Queued or Running and also you will get a success message alert at the top of your page saying that Job was successfully submitted. You will see the status Completed after the job execution gets finished.

    Modify Job Script
  5. To view the output after the job finishes, click the generated output file slurm-<job-id>.out under Folder contents in Job Details section.

    Default Job Output

    Output(slurm-<job-id>.out):

    Default Job Output

From Template

Instead of retyping the Slurm attributes and job parameters for your new job, you can create a custom template and use as a basis for your future jobs. It also saves time. Also, it’s easier and faster to create a custom template in Open OnDemand. Follow the below steps to create a custom template and compose job from that template.

  1. First, create a directory called custom_template under your home directory.

    [crushev@discovery-l2 ~]$ mkdir custom_template
  2. Create the following files, input.txt, script.sh under custom_template directory.

    • input.txt

    • script.sh

    Ubuntu,apt
    Debian,apt
    CentOS,yum
    Arch Linux,pacman
    Fedora,dnf
    #!/bin/bash
    
    while IFS= read -r line; do
      if [[ "$line" == *"apt"* ]]; then
        printf '%s\n' "$line"
      fi
    done < input.txt

    The bash script script.sh will read the input file line by line and if it finds a line that has the string apt, then, it will print the line.

  3. The custom template is ready and available under the path <home></your-username></custom_template>. Now, using Open OnDemand, you can create a template.

    1. Go to Jobs > Jobs Composer from the dashboard. Select the templates tab.

      Create Template
    2. Select New Template from the Templates page.

      New Template
    3. Now, you can enter the values for the custom template. Enter the path and name to the template and hit Save button.

      New Template
    4. You will see the template has been successfully created and you will see a similar output like below.

      Custom Template Output

      Also, on the right bottom, you can see Template location and also Folder Contents with the files input.txt, script.sh and an additional file called manifest.yml.

  4. Now, you can create a job from the template. Select the Create New Job button from the Templates page. A new job will be created using the custom template and you will be automatically redirected to the Jobs page.

    Custom Template - Create Job
  5. Submit the job by clicking Submit button and you will the status change from Not Submitted to Queued or Running. You will also see a success alert for the job submission at the top of your page.

    Custom Template - Submit Job
  6. After the status changes to completed which indicates that the job has finished, select the output file slurm-<job-id>.out under the folder contents section on the right pane of the window.

    Template Job Output

    Output(slurm-<job-id>.out):

    Template View Output

From Specified Path

Creating jobs from Specified Path is one of the simplest ways of creating a job but there are things you need to take note of.

While creating a job from a Specified Path, make sure that your scripts are present in a directory regardless of the location.

If you have a submission script called script.sh in your home directory and you specify path as home/<your-username>, it will cause all the content such as files and sub directories in the home directory to be copied under the new workspace which is a bad and space overkill. It’s recommended to create a directory and place all the necessary scripts under that directory and specify it as a source path for your jobs.

  1. First, create a directory called test under your home directory.

    [crushev@discovery-l2 ~]$ mkdir test
  2. Create the following files, script.py, script.sh under test directory.

    • script.sh

    • script.py

    #!/bin/bash
    
    #SBATCH --job-name=maxFib      ## Name of the job
    #SBATCH --output=maxFib.out    ## Output file
    #SBATCH --time=0-00:10:00      ## Job Duration
    #SBATCH --ntasks=1             ## Number of tasks (analyses) to run
    #SBATCH --cpus-per-task=1      ## The number of threads the code will use
    #SBATCH --mem-per-cpu=100M     ## Real memory(MB) per CPU required by the job.
    
    ## Load the python interpreter
    module load python
    
    ## Execute the python script and pass the argument/input '90'
    srun --export=ALL python script.py 90
    import sys
    import os
    
    if len(sys.argv) != 2:
        print('Usage: %s MAXIMUM' % (os.path.basename(sys.argv[0])))
        sys.exit(1)
    
    maximum = int(sys.argv[1])
    
    n1 = 1
    n2 = 1
    
    while n2 <= maximum:
        n1, n2 = n2, n1 + n2
    
    print('The greatest Fibonacci number up to %d is %d' % (maximum, n1))

    If you have a close look at the submit script script.sh, it has --export=ALL flag in the srun command. The flag is mandatory if you want to submit and run jobs to the Discovery cluster through Open OnDemand.

  3. Now, navigate to Jobs > Job Composer. Select New Job > From Specified Path.

    From Specified Path
  4. Enter the source path to test directory and give a name for the job as PythonJob. Also, give a name for the submission script as script.sh and specify the cluster name as Discovery. Hit Save button after setting all the job options.

    Set Values
  5. You will see a new job created in the Jobs page. Now, select the Submit button to launch the job.

    Submit Job
  6. Now, you will see the status changing from Not Submitted to Queued/Running. After the status changes to Completed, you can view the output maxFib.out under Folder Contents on the right pane.

    Job Output

    Output(maxFib.out):

    View Output

From Selected Job

  1. To create a new job from the existing one, select New Job > From Selected Job.

    Create Selected Job
  2. You will have a new job(copy) created in the Jobs page.

    Selected Job Output

You can also use the buttons Stop, Delete to stop a running job and delete the selected job respectively.