File Management

Different ways of file management in Discovery are discussed below.

1. Open OnDemand

One of the easiest ways for managing the files and directories in Discovery is through Open OnDemand portal. Open OnDemand provides a web interface which allows for easy management of files and directories in Discovery. More detailed documentation about file management using Open OnDemand can be found here → File Manager - Open OnDemand.

Using Open OnDemand, you can upload files of size up to 10 GB. If you want to upload files of size >10 GB, you can do SCP or SFTP file transfers to discovery which are discussed in the below sections.

2. SCP File Access

  • If you want to access Discovery and manage files via SCP, you require NMSU VPN access if you’re outside of the NMSU network. Read the VPN information at the link → NMSU VPN Access.

Once connected to the NMSU network, you can now transfer files to Discovery. To upload/download the files to and from Discovery, scp command can be used.

SCP Command

  • It stands for Secure Copy Protocol.

  • It uses SSH for file transfers and hence it requires SSH key or password authentication to connect to the remote systems.

  • SCP is included in Mac and Linux by default. For Windows Users, it’s included in Windows 10(From April 2018 release) and Windows 11 by default.

  • The syntax for the scp command:

    scp [OPTIONS] [SOURCE FILE PATH] [DESTINATION FILE PATH]
    Parameter Description

    [OPTIONS]

    Allows to specify set of options like such as cipher, SSH configuration, etc.

    [SOURCE FILE PATH]

    Path to the Source File.

    [DESTINATION FILE PATH]

    Path to the Destination File.

Run the man scp command from the terminal to display the user manual of the command.

Uploading Files (Local Workstation to Remote Discovery)

Uploading a File

  • To upload/copy a file called example.txt under the home directory from your local system to your home directory in Discovery, enter the following command from the terminal.

    scp ~/example.txt <username>@discovery.nmsu.edu:/home/<username>

    Shell Output

    Password:
  • You’ll be asked to enter the password associated with your Discovery account. Only on successful authentication, you will be able to transfer the files.

    Shell Output

    example.txt                                   100%    9     1.2KB/s   00:00

Uploading a Directory

  • To upload/copy a directory called test placed under the home directory from your local machine , you simply need to add the -r flag like below.

    Syntax

    scp -r <local-directory-path> <remote-username>@discovery.nmsu.edu:<remote-destination-path>

    Example

    scp -r ~/test <username>@discovery.nmsu.edu:/home/<username>
  • You will be asked to enter the password of your Discovery account. On successful authentication, your directory will be copied successfully.

Downloading Files (From Discovery to Local System)

  • The syntax for downloading files from Discovery to local system is as follows:

    Syntax:

    scp <username>@discovery.nmsu.edu:<remote-file-path> <destination(local)-path>
  • Run the below command to download/copy the file called example.txt from your home directory in Discovery to your local home directory.

    scp <username>@discovery.nmsu.edu:/home/<username>/example.txt ~
  • You will be asked to enter the password and the file example.txt will be copied to your local home directory successfully.

  • To copy/download a directory called test from your home directory in Discovery to your local home directory, simply use the -r option like below.

    scp -r <username>@discovery.nmsu.edu:/home/<username>/test ~

3. SFTP File Access

  • SFTP(SSH File Transfer Protocol) is a secure file protocol that’s used to manage files over an encrypted SSH transport.

  • SFTP offers additional functionalities over SCP like performing a range of operations on remote files like viewing directories, deleting files, directories, and much more.

  • SFTP is included in Mac and Linux by default. For Windows Users, it’s included in Windows 10(From April 2018 release) and Windows 11 by default.

    • If you want to access Discovery and manage files via SFTP, you require NMSU VPN access if you’re outside of the NMSU network. Read the VPN information at the link → NMSU VPN Access.

Connect to Discovery via SFTP

  • Open your terminal and type the following command. You need to enter the username associated with your Discovery account.

    Syntax

    sftp <username>@discovery.nmsu.edu
  • Then, you will be asked to enter the password. On successful authentication, you will be connected to Discovery and see the sftp> shell prompt like below.

    Output

    Authorized uses only. All activity may be monitored and reported.
    Password:
    Connected to discovery.nmsu.edu.
    sftp>

List Files

  • To list the files on the remote system, run the following command.

    Syntax

    sftp> ls
  • To list the files on the local system, run the following command.

    Syntax

    sftp> lls

Print the Current Directory

  • To display the current working directory of the remote system, run the following command.

    Syntax

    sftp> pwd
  • To display the current working directory of the local system, run the following command.

    Syntax

    sftp> lpwd

Change Working Directory in Remote System

  • To change the working directory of the remote machine, run the following command.

    Syntax

    sftp> cd <remote-path>

    Example

    sftp> cd /scratch/<username>
  • To change the working directory of the local system, run the following command.

    Syntax

    sftp> lcd <path>

    Example

    lcd /Users/<username>/Downloads

Create Directory

  • To create a remote directory , the syntax is as follows:

    Syntax

    sftp> mkdir <remote-directory-path>

    Example

    sftp> mkdir /scratch/<username>/test
  • A new directoy called test will be created under the scratch directory.

  • To create a directory on your local system, run the following command:

    Syntax

    sftp> lmkdir <directory-name>

    Example

    sftp> lmkdir test

Remove a Remote Directory/File

  • To remove a remote directory , the syntax is as follows:

    Syntax

    sftp> rmdir <remote-directory-path>

    Example

    sftp> rmdir /scratch/<username>/test
  • To remove a remote file, the syntax is as follows:

    Syntax

    sftp> rm <remote-file-path>

    Example

    sftp> rm /scratch/<username>/test.txt

Upload a File/Directory

  • To upload a file called example.txt from your local system’s working directory to remote /home/<username> directory in Discovery, run the following command.

    Syntax

    sftp> put <filename>

    Example

    sftp> put example.txt

    Output

    Uploading example.txt to /scratch/<remote-username>/example.txt
    example.txt                                               100%    0     0.0KB/s   00:00
    sftp>

    To upload a file which isn’t present in the current working directory of your local system, enter the absolute path of the file.

  • To upload a directory called test from your local system’s working directory to remote directory in Discovery, run the following command:

    Syntax

    sftp> put -r <directory-name>

    Example

    sftp> put -r test

    Output

    Uploading test/ to /home/<username>/test
    Entering test/
    test/test1.txt                                100%    0     0.0KB/s   00:00
    test/test2.txt                                100%    0     0.0KB/s   00:00
    sftp>

Download a File/Directory

  • Similarly, to download a file called example.txt to your local system’s working directory from the remote working directory in Discovery, you can use the get command like below:

    Syntax

    sftp> get <filename>

    Example

    sftp> get example.txt

    Output

    Fetching /home/<username>/example.txt to example.txt
    /home/<username>/example.txt                     100%    9     0.0KB/s   00:00
    sftp>
  • To download a directory called test to your local system’s working directory from the remote working directory in Discovery, run the following get command with -r flag.

    Syntax

    sftp> get -r <directory-name>

    Example

    sftp> get -r test

    Output

    Fetching /home/<username>/test/ to test
    Retrieving /home/<username>/test
    sftp>

Available Commands

  • To get the list of available commands, run the following help or ? command from the sftp shell prompt.

    Syntax

    sftp> help

    Output

    bye                                Quit sftp
    cd path                            Change remote directory to 'path'
    chgrp [-h] grp path                Change group of file 'path' to 'grp'
    chmod [-h] mode path               Change permissions of file 'path' to 'mode'
    chown [-h] own path                Change owner of file 'path' to 'own'
    df [-hi] [path]                    Display statistics for current directory or
                                       filesystem containing 'path'
    exit                               Quit sftp
    get [-afpR] remote [local]         Download file
    help                               Display this help text
    lcd path                           Change local directory to 'path'
    lls [ls-options [path]]            Display local directory listing
    lmkdir path                        Create local directory
    ln [-s] oldpath newpath            Link remote file (-s for symlink)
    lpwd                               Print local working directory
    ls [-1afhlnrSt] [path]             Display remote directory listing
    lumask umask                       Set local umask to 'umask'
    mkdir path                         Create remote directory
    progress                           Toggle display of progress meter
    put [-afpR] local [remote]         Upload file
    pwd                                Display remote working directory
    quit                               Quit sftp
    reget [-fpR] remote [local]        Resume download file
    rename oldpath newpath             Rename remote file
    reput [-fpR] local [remote]        Resume upload file
    rm path                            Delete remote file
    rmdir path                         Remove remote directory
    symlink oldpath newpath            Symlink remote file
    version                            Show SFTP version
    !command                           Execute 'command' in local shell
    !                                  Escape to local shell
    ?                                  Synonym for help

Quitting SFTP Shell

  • To quit the SFTP Shell and return to your local system’s terminal, run the bye or quit command.

    Syntax

    sftp> bye|quit

4. WinSCP Software (Windows Users)

  • For Windows 10(From April 2018) and Windows 11, SCP and SFTP are available by default and you can use PowerShell/Command Prompt to perform the file transfers using the above discussed SCP and SFTP methods.

  • If your windows PowerShell or Command Prompt isn’t configured/supports to perform SCP or SFTP file transfers(Windows versions before April 2018), there are several free and open-source GUI software available and you can install one of those and transfer files.

  • One of the recommended and popular software called WinSCP is available for Windows Users and it enables easy file management between local and remote system.

  • To learn more about WinSCP, refer to the documentation → WinSCP.