Introduction

This guide explains how you can run Jupyter Notebook on the Discovery cluster.

This can be carried out in three different ways:

  • Using Open OnDemand

  • Running the web app interactively with srun.

  • Submitting a batch script with SBATCH

Running Jupyter Notebook

Using Open OnDemand(Easier and Faster way)

  • Open OnDemand provides an easier and faster way of launching the Jupyter Notebook through an interactive session on the compute node.

  • Users simply need to fill a form in OnDemand web portal to request the resources and access the Jupyter notebook, thus eliminating the need of running a series of bash commands to launch Jupyter notebook from the terminal.

  • For more information on how to launch Jupyter Notebook via Open OnDemand, refer the link → Launching Jupyter Notebook using Open OnDemand.

Using srun (Interactively)

  1. Open a new terminal session and connect to Discovery

  2. Launch tmux.

    tmux

    Starting srun within tmux is recommended because it helps persist the program’s activity even when your connection to Discovery’s login node is terminated. For more handy commands visit → tmux cheatsheet website.

  3. Load Anaconda

    module purge
    module load anaconda
  4. Execute the srun command with the following parameters

    srun --time=02:00:00 --ntasks=1 --cpus-per-task=2 --mem-per-cpu 4g jupyter notebook --no-browser --ip=0.0.0.0

    Flag

    Description

    --ntasks

    Notice the value of ntasks is 1. Specifying more than one ntasks may not work for Jupyter notebook.

    --no-browser

    Prevents the browser window from launching on the remote server.

    --ip

    Set the default network.

    jupyter notebook

    The web application to launch.

    In less than 5 seconds you will get an output similar to the one below on your console.

    [I 19:07:44.422 NotebookApp] Writing notebook server cookie secret to /home/vaduaka/.local/share/jupyter/runtime/notebook_cookie_secret
     [I 19:07:48.089 NotebookApp] JupyterLab extension loaded from /software/anaconda/anaconda3/lib/python3.7/site-packages/jupyterlab
     [I 19:07:48.089 NotebookApp] JupyterLab application directory is /software/anaconda/anaconda3/share/jupyter/lab
     [I 19:07:48.094 NotebookApp] Serving notebooks from local directory: /home/vaduaka
     [I 19:07:48.094 NotebookApp] The Jupyter Notebook is running at:
     [I 19:07:48.094 NotebookApp] http://discovery-c7.cluster.local:8888/?token=8149d1db4034d604e8681f457c3e23fef2f0b5dccab1b413
     [I 19:07:48.094 NotebookApp]  or http://127.0.0.1:8888/?token=8149d1db4034d604e8681f457c3e23fef2f0b5dccab1b413
     [I 19:07:48.094 NotebookApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).
     [C 19:07:48.101 NotebookApp]
    
         To access the notebook, open this file in a browser:
             file:///home/vaduaka/.local/share/jupyter/runtime/nbserver-47967-open.html
         Or copy and paste one of these URLs:
             http://discovery-c7.cluster.local:8888/?token=8149d1db4034d604e8681f457c3e23fef2f0b5dccab1b413
          or http://127.0.0.1:8888/?token=8149d1db4034d604e8681f457c3e23fef2f0b5dccab1b413

    From the output above you can see that Jupyter notebook is running on host: discovery-c7.cluster.local port: 8888 and the generated token:8149d1db4034d604e8681f457c3e23fef2f0b5dccab1b413

    At this point neither http://127.0.0.1:8888 nor http://discovery-c7.cluster.local:8888 addresses would work. This is because the connection needs to be tunneled through the login node and redirect the port.

  5. To access Jupyter notebook’s web interface, open a new terminal session and connect to Discovery.

    1. Open a new terminal

    2. Connect to Discovery via SSH with the following parameters

      ssh -N -L 8888:discovery-c7:8888 yourusername@discovery.nmsu.edu

      Basically, the -N flag is required for just forwarding ports and prevents the shell from executing a remote command. The -L flag specifies that the given port 8888: on the local (client) host is to be forwarded to discovery-c7 on port :8888 via Discovery’s login node discovery.nmsu.edu.

    3. On your client machine, open your browser(Chrome, Firefox, IE), go to the following → http://localhost:8888 or http://127.0.0.1:8888 and you will be presented with Jupyter notebook’s login page.

    4. Now copy and paste the generated token from the previous terminal (from step 4) and paste into the password or token text box and click log in. Copying and pasting the full address(with the token ID) http://127.0.0.1:8888/?token=8149d1db4034d604e8681f457c3e23fef2f0b5dccab1b413 bypasses the log in page and signs you in automatically.

      running-web-applications1.png
    5. Next, Open and run the Jupyter notebook project file located in my home directory.

      running-web-applications2.png
      running-web-applications3.png
  6. Logout: Next, when you are done interacting with Jupyter notebook and want to end the session, please navigate to the Jupyter notebook’s home page and click on the Quit button to shutdown the Jupyter server and kill the running job.

    running-web-applications4.png
    running-web-applications3.png

    Finally, on the second terminal use Ctrl + C to quit the port forwarding.

    It’s crucial that you use the Quit button rather than the Logout button to kill your interactive session with Jupyter notebook. Otherwise, you’d still be consuming resources on the Discovery cluster even when you are done with your intended task.

Using SBATCH script

  1. Open a new terminal session and connect to Discovery

  2. Create a new batch script and make it executable

    vi script.sh && chmod +x script.sh
  3. Copy, paste and save the script below in the script.sh file

    #!/bin/bash
    #SBATCH --job-name=jupyterTest
    #SBATCH --ntasks=1
    #SBATCH --cpus-per-task=2
    #SBATCH --mem-per-cpu=1g
    #SBATCH --time=2:00:00
    
    # Load necessary modules
    module purge
    module load anaconda
    
    # Start the notebook
    srun jupyter notebook --no-browser --ip=0.0.0.0
  4. Submit the batch script

    sbatch script.sh

    Output

    Submitted batch job 115243
  5. Check output file for the node and port information.

    tail -f slurm-115243.out

    Output

    ...
    
        To access the notebook, open this file in a browser:
         file:///home/vaduaka/.local/share/jupyter/runtime/nbserver-21312-open.html
     Or copy and paste one of these URLs:
         http://discovery-c1.cluster.local:8888/?token=d0729c0a802e08856109ef837b871fb6ddfd5e5791453e83
      or http://127.0.0.1:8888/?token=d0729c0a802e08856109ef837b871fb6ddfd5e5791453e83
    
    ...

    Now take note of the host: discovery-c1.cluster.local port: 8888 and the generated token:d0729c0a802e08856109ef837b871fb6ddfd5e5791453e83

  6. Finally, carry out step 5 process of Running Jupyter Notebook - Using srun (Interactively) and replace the host: discovery-c1, port: 8888 and token: d0729c0a802e08856109ef837b871fb6ddfd5e5791453e83 where necessary.