Storage – IPPP
Local Scratch
Every system has a local partition available for writing to, this is /scratch.
We recommend putting your data into a sub-directory with your username (this may already be generated on some systems) this is to stop other users editing/removing your data.
mkdir /scratch/$USERNAME && chmod 700 /scratch/$USERNAME
We recommend using scratch as much as possible as this will always be faster than any shared network storage.
Scratch space is not shared across the network and so is only available locally to that individual system, so if you require the data over many systems, please ensure you copy this to and from a shared filesystem prior/end of your work.
Scratch is never backed up and is liable to be erased without notice as soon as a system is faulty although we try to keep any loss to a minimum.
Please note that the scratch is designed for research purposes only and not to store sensitive / Personally Identifiable Information (PII), we do not encrypt this data.
Shared Network
Home
Your home space is on a shared network server and is automatically mapped to you ( ~/ and /mt/home/$USERNAME ). Your home directory is restricted access to only yourself.
By default all users get 30GB, we can expand this upon request up to 75GB. It is backed up daily.
The home server is used by everyone all the time, so we request that you do not run batch jobs, jupyter notebooks or any significant IO heavy work from/to your home directory.
Please note that the system is designed for research purposes only and not to store sensitive / Personally Identifiable Information (PII), we do not encrypt this data or encrypt the backups.
User Batch
User Batch is limited to a single user access. i.e. you cant let another account access this data.
User Batch space (or “User Batch” ) is available at /mt/user-batch/$USERNAME, this storage is aimed at your IO heavy work that requires personal storage, we generally recommend that you copy (ideally tar’d) your data to/from rather than running directly from Batch but we are aware that this is not always possible.
By default, users get 1TB, there is currently limited expansion upon request all the way up to 10TB depending on the current overall utilisation.
User Batch is NOT backed up, the amount of data stored is far too great to economically backup, so please be aware that data loss on this system is permanent.
Please note that the system is designed for research purposes only and not to store sensitive / Personally Identifiable Information (PII), we do not encrypt this data.
Group Batch
Group Batch is space that allows multiple users to access (read and write) the data. This must be requested, so contact us for a share to be created.
Group Batch space (or “Group Batch” ) is available at /mt/group-batch/$GROUPNAME-share, this storage is aimed at your IO heavy work that requires shared storage, we generally recommend that you copy (ideally tar’d) your data to/from rather than running directly from Batch..
By default, groups get a shared 1TB, there is currently limited expansion upon request all the way up to 10TB depending on the current overall utilisation.
Group Batch is NOT backed up, the amount of data stored is far too great to economically backup, so please be aware that data loss on this system is permanent.
As these directories are shared amongst a group of users please be vigilant of the file permissions you set to ensure data is not incorrectly overwritten. Please see the tutorial at Linux Foundation for more help. SetUID and SetGUID do not apply.
Please note that the system is designed for research purposes only and not to store sensitive / Personally Identifiable Information (PII), we do not encrypt this data.
Other Useful Storage
Personal Webspace
Your home contains a directory called www ( ~/www ), this is directly mapped on our publicly accessible web server. Anything put into here will be available via https://www.ippp.dur.ac.uk/~$USERNAME/
Pre-Compiled HEP Software
We have a small repository of pre-compiled HEP Software for you to use at /mt/hepsw/
Seafile Cloud Storage
See our Cloud Storage page for information on Seafile or also historically known as “Laptop Backups”.
Quota
To find out your storage usage/quota on Home and User Batch please use the zquota command and you should see an output similar to below
login~$ zquota Quota Report for username Mount Point Used Total Last Checked /mt/home 14G (9%) 150G Mon 12 Apr 2021 12:22:06 BST /mt/user-batch 127M (0%) 1.0T Mon 12 Apr 2021 12:22:03 BST
The zquota command will now also display any group batch storage usage, this will show the entire group usage and quota and not your individual allocation.
Anti-Virus Scanning
We periodically scan your file storage allocation for viruses. We receive a notification as soon as a virus is detected so we can apply the appropriate action, by default the system will not remove anything it suspects is a virus but we may do so manually.
A log of any files suspected to be a virus can be found in your home space under ~/.virusscan or /mt/user-batch/$USERNAME/.virusscan.