Sharing Files On The HPC Cluster

Introduction

On the HPC, when you collaborate with other users on a project, you will occasionally wish to share files between project members. There are several ways to do this, often determined by how often and for how long sharing is needed.

Leveraging a group is a convenient way for users to share files repeatedly over an extended period.

To use this method, request a project group and directory by email. Send the following information:

A reason for the request
A group name that meets the following criteria:
1. It should be short (up to some eight (8) characters),
2. It should be descriptive, and
3. It should be distinctive.
The initial user list (later changes are simple to request).

Once UITS receives the request, they'll create a new group with your requested members. If you are connected to the HPC when the group is created, you'll need to log out and log in to join the group.

A group directory will be created and can be accessed by one of the following names:
```
/data/share/groupname
~groupname
```
The directory will be mutually readable and writable among group members and will not be accessible by other users.
The group will usually be inherited by all files and directories created there.

Tip

Fortran programs typically cannot interpret the ~groupname shortcut form of the share directory. Give the full path or cd into the directory before running the program, then use relative path names.

Granting Write Permission

The directories in /data/share are configured such that users of a group can read and write files and directories. However, when you move existing directories and files from somewhere else into the share directory, they may not be writable for the group.

To make files writeable by group members
```
[barney@hpc ~]$ chmod g+w file
```
To make a directory and all files thereunder (-R) group writable, which includes the ability to delete files:
```
[barney@hpc ~]$ chmod -R g+w dirname
```

Learn more about chmod, chown, and chgrp.

You can give access to some of your directories in your home or scratch directories by opening permissions.

This option is typically useful for one-way sharing.
For bidirectional sharing, you will need to grant world-write permissions, which is not recommended for files in your own directory space.
You may need to open permissions whenever anything is added to the directory.

You, as the owner, can arrange this on your own as follows:

[barney@hpc ~]$ chmod a+x ${HOME}/
[barney@hpc ~]$ mkdir ${HOME}/share
# put new content into your share...
[barney@hpc ~]$ chmod -R a+rX ${HOME}/share

This will yield read ("r") and execute ("X") permissions for all users ("a"), meaning users can browse directories, and read and execute already executable files there. You may have to repeat the last step after you add files in the shared directory.

Direct your fellow users as follows:

[barney@hpc ~]$ cd /data/home/userid/share
[barney@hpc ~]$ ls
[barney@hpc ~]$ cp -p ....

The first cd is crucial to have your collaborator step into a sharing directory you designated instead of stepping in from your home directory. Other users are not usually able to simply rummage around your home directory with ls -- it is up to you to show them a specific directory they can browse.

Using a /tmp Directory

You can drop files into a login node's general-use /tmp directory and set permissions as needed. This way of sharing is:

suitable for one-off exchanges,
typically fairly loose with permissions,
resides on only one host,
world-write is not recommended,
likely to need chmod whenever more files are added.

To share files under /tmp, do the following:

[barney@hpc ~]$ mkdir /tmp/foo
# put files and directories in your temp directory
[barney@hpc ~]$ chmod -R a+rX /tmp/foo

References

Adapted from the Argonne National Laboratory Wiki page: https://wiki.anl.gov/cnm/HPC/Sharing_Files