Storage – IPPP

Local Scratch

Every system has a local partition available for writing to, this is /scratch.

We recommend putting your data into a sub-directory with your username (this may already be generated on some systems) this is to stop other users editing/removing your data.

mkdir /scratch/$USERNAME && chmod 700 /scratch/$USERNAME

We recommend using scratch as much as possible as this will always be faster than any shared network storage.

Scratch space is not shared across the network and so is only available locally to that individual system, so if you require the data over many systems, please ensure you copy this to and from a shared filesystem prior/end of your work.

Scratch is never backed up and is liable to be erased without notice as soon as a system is faulty although we try to keep any loss to a minimum.

Shared Network

Home

Your home space is on a shared network server and is automatically mapped to you ( ~/ and /mt/home/$USERNAME ).

By default all users get 30GB, we can expand this upon request up to 75GB. It is backed up daily.

The home server is used by everyone all the time, so we request that you do not run batch jobs, jupyter notebooks or any significant IO heavy work from/to your home directory.

User Batch

User Batch is limited to a single user access. i.e. you cant let another account access this data.

User Batch space (or “User Batch” ) is available at /mt/user-batch/$USERNAME, this storage is aimed at your IO heavy work that requires personal storage, we generally recommend that you copy (ideally tar’d) your data to/from rather than running directly from Batch but we are aware that this is not always possible.

By default, users get 1TB, there is currently limited expansion upon request all the way up to 10TB depending on the current overall utilisation.

User Batch is NOT backed up, the amount of data stored is far too great to economically backup, so please be aware that data loss on this system is permanent.

Group Batch

Group Batch is space that allows multiple users to access (read and write) the data. This must be requested, so contact us for a share to be created.

Group Batch space (or “Group Batch” ) is available at /mt/group-batch/$GROUPNAME-share, this storage is aimed at your IO heavy work that requires shared storage, we generally recommend that you copy (ideally tar’d) your data to/from rather than running directly from Batch..

By default, groups get a shared 1TB, there is currently limited expansion upon request all the way up to 10TB depending on the current overall utilisation.

Group Batch is NOT backed up, the amount of data stored is far too great to economically backup, so please be aware that data loss on this system is permanent.

As these directories are shared amongst a group of users please be vigilant of the file permissions you set to ensure data is not incorrectly overwritten. Please see the tutorial at Linux Foundation for more help. SetUID and SetGUID do not apply.

Other Useful Storage

Personal Webspace

Your home contains a directory called www ( ~/www ), this is directly mapped on our publicly accessible webserver. Anything put into here will be available via https://www.ippp.dur.ac.uk/~$USERNAME/

Pre-Compiled HEP Software

We have a small repository of pre-compiled HEP Software for you to use at /mt/hepsw/

Seafile Cloud Storage

See our Cloud Storage page for information on Seafile or also historically known as “Laptop Backups”.

Batch

Batch was removed on the 15th March 2021. Please see “User Batch” and “Group Batch” above.

Quota

To find out your storage usage/quota on Home and User Batch please use the zquota command and you should see an output similar to below

login~$ zquota

Quota Report for username
Mount Point Used Total Last Checked
/mt/home 14G (9%) 150G Mon 12 Apr 2021 12:22:06 BST
/mt/user-batch 127M (0%) 1.0T Mon 12 Apr 2021 12:22:03 BST

The zquota command will now also display any group batch storage usage, this will show the entire group usage and quota and not your individual allocation.

Anti-Virus Scanning

We periodically scan your file storage allocation for viruses. We recieve a notification as soon as a virus is detected so we can apply the appropriate action, by default the system will not remove anything it suspects is a virus.

A log of any files suspected to be a virus can be found in your home space under ~/.clamscan or /mt/user-batch/$USERNAME/.clamscan.