Table of Contents

Storage – Grid

You are here:
< All Topics

To use the grid storage you must first have a grid certificate.

Various storage systems are available which should enable you to run your jobs and efficiently scale them.

SE – Storage Element aka DPM

Namese01.dur.scotgrid.ac.uk
Protocolsgsiftp, xroot, davs (file)
Storage Available~1PB

The SE at Durham is an object file system that utilises DPM/DMLite. There are several protocols available which are listed above. We highly recommend that people utilise the SE for all Jobs that exceeds 10MB in either input or output files or both combined.

You will find that although this is an extra layer of complexity, the speed, capacity and robustness is worth the effort. DPM is constantly developing and changing so if you’re struggling get in touch.

GFAL2 Toolset

The gfal2 toolset should feel relatively familiar to the standard Linux file tools. Be aware that if you use the file:// handler for local files, you need to include the full directory path. Also be aware that not all sites conform to the same standard.

Make a Directory

To make a directory

gfal-mkdir <protocol>://<se_address>/dpm/<site_domain>/home/<vo>/<username>

For example

gfal-mkdir gsiftp://se01.dur.scotgrid.ac.uk/dpm/dur.scotgrid.ac.uk/home/pheno/MyUserName

View a Directory

To view a directory

gfal-ls <protocol>://<se_address>/dpm/<site_domain>/home/<vo>/<username>

For example

gfal-ls gsiftp://se01.dur.scotgrid.ac.uk/dpm/dur.scotgrid.ac.uk/home/pheno/MyUserName

Copy a File

To copy a file

gfal-copy <protocol>://<file_source> <protocol>://<file_source>

For example to copy a file from the SE

gfal-copy gsiftp://se01.dur.scotgrid.ac.uk/dpm/dur.scotgrid.ac.uk/home/pheno/MyUserName/MyAwesomeFile file://$(pwd)/MyAwesomeFile

For example to copy a file to the SE

gfal-copy file://$(pwd)/MyAwesomeFile gsiftp://se01.dur.scotgrid.ac.uk/dpm/dur.scotgrid.ac.uk/home/pheno/MyAwesomeFile

Remove a File or Directory

To Remove things; Just like rm you need to use -r for directories, there is also a –dry-run option to check what it will do before hand.

gfal-rm <protocol>://<se_address>/dpm/<site_domain>/home/<vo>/<username>/<file_or_dir>

For example

gfal-rm gsiftp://se01.dur.scotgrid.ac.uk/dpm/dur.scotgrid.ac.uk/home/pheno/MyUserName/MyAwesomeFile

Stream File to StdOut

To stream a file via stdout from the SE (Useful for file chunking)

gfal-cat <protocol>://<se_address>/dpm/<site_domain>/home/<vo>/<username>/<filename> > ~/<filename>

Stream StdIn to File

To stream from stdin to the SE (Useful for outputs)

cat /etc/os-release | gfal-save <protocol>://<se_address>/dpm/<site_domain>/home/<vo>/<username>/os-release

LFC – Logical File Catalog

This system is being phased out due to lack of official support and we can no longer support it. It is only mentioned here for documentation reasons.

Namelfc01.dur.scotgrid.ac.uk
Sites AvailableDurham

GFAL2 Toolset

Please use the gfal suite and target the LFC as well as the gfal-legacy-* commands.

CVMFS – Cern Virtual Machine File System

This is a read only HTTP file system that you can interact with as though it’s a standard local file system, it is available across all GridUIs and nodes that support GridPP work, if a site supports Pheno it should have the Pheno directory available. Each VO have their own directories.

Pheno can be found in /cvmfs/pheno.egi.eu

If you want write access to this then please contact us as it is restricted access and a complex method and we don’t explain it on a public guide.

Previous Storage – General