Quotas and Folders

Quotas in Discovery

Quotas are the disk space available for the HPC users to use it in their computational tasks. Whenever you log in to the Discovery, you get the following report about the quotas.

## Quota Reports ##
/home/crushev                   Used=18.8G   Quota=100G
/scratch/crushev                Used=252M    Quota=1T
/fs1/project/hpcteam            Used=327.8G  Quota=5T
/fs1/project/ml-classification  Used=107.9M  Quota=500G

  • In home folder, you have a Quota of 100GB and the Used parameter shows the currently used disk space by the user in this folder. The scratch folder has a Quota of 1TB.

  • Also, if you have access to any shared project space, the quota information for project folders will also be shown(like above).

  • You can also run the checkquota.sh command from the login node to check the quota details.

Syntax

checkquota.sh

Output

## Quota Reports ##
/home/crushev                   Used=18.92G  Quota=100G
/scratch/crushev                Used=252M    Quota=1T
/fs1/project/hpcteam            Used=327.8G  Quota=5T
/fs1/project/ml-classification  Used=107.9M  Quota=500G

  • Also, if you have access to any shared project space, the quota information for project folders will also be shown(like above).

Syntax

checkquota.sh

Output

## Quota Reports ##
/home/crushev                   Used=18.8G   Quota=100G
/scratch/crushev                Used=252M    Quota=1T
/fs1/project/hpcteam            Used=327.8G  Quota=5T
/fs1/project/ml-classification  Used=107.9M  Quota=500G

Folders in Discovery

Some important folders in Discovery that you should be aware of are as follows:

/home

It’s the default directory or the working environment provided by the Linux. Whenever you login to the Discovery cluster, you are in the home/<your-username> folder. It’s recommended to store data which is relatively small in the home folder because you have only 100GB of storage. This folder is meant for long-term storage of data.

Run the pwd command to show the current working directory.

/scratch

The scratch folder is ideal for jobs that has a lot of reading/writing tasks and involves a lot of disk space. However, /scratch folder isn’t backed up by the HPC team and hence it’s advisable to place only the temporary files in it. The files present in this directory gets deleted after some time if remain unaccessed. The reason behind the large storage space in scratch folder is to allow the users to work with the large datasets. Also, it allows to have a working environment that can handle huge volumes of data.

Refer to the section → Backups and Data retention in Discovery in this page below to learn more about the backups.

/project

Users working together on a project can request for 500GB by default. This storage space will be shared by the users working on the project. The request for this project space can be made by the faculty by submitting the form in the link → Project Space Request Form . Once approved, you will find the created project space under this folder.

The permissions on the project directory will be set as owner=root, group="group of users who share that space"

Backups and data retention in Discovery

HPC team make nightly backups of /home and /project and the backups are kept for 30 days. Files in /home and /project will be kept as long as your account is active. If your account remains inactive for 1 year, the account will get locked and after 60 days of grace period, your files in /home and /project will be deleted.

HPC team don’t make backups of /scratch. Files in /scratch which aren’t accessed for 120 days will be automatically deleted. Users will get notifications when their files weren’t accessed for 92 days, 106 days, and 120 days(purged).

What’s next if the storage space is exhausted?

If you have exhausted the storage space, you can request for extra space in /home and /project folders by submitting the form in this link → Storage Request Form. However, the extra space comes at an extra cost.