Storage – Grid
To use the grid storage you must first have a grid certificate.
Various storage systems are available which should enable you to run your jobs and efficiently scale them.
SE – Storage Element aka DPM
|Protocols||gsiftp, xroot, davs (file)|
The SE at Durham is an object file system that utilises DPM/DMLite. There are several protocols available which are listed above. We highly recommend that people utilise the SE for all Jobs that exceeds 10MB in either input or output files or both combined.
You will find that although this is an extra layer of complexity, the speed, capacity and robustness is worth the effort. DPM is constantly developing and changing so if you’re struggling get in touch.
The gfal2 toolset should feel relatively familiar to the standard Linux file tools. Be aware that if you use the file:// handler for local files, you need to include the full directory path. Also be aware that not all sites conform to the same standard.
Make a Directory
To make a directory
View a Directory
To view a directory
Copy a File
To copy a file
gfal-copy <protocol>://<file_source> <protocol>://<file_source>
For example to copy a file from the SE
gfal-copy gsiftp://se01.dur.scotgrid.ac.uk/dpm/dur.scotgrid.ac.uk/home/pheno/MyUserName/MyAwesomeFile file://$(pwd)/MyAwesomeFile
For example to copy a file to the SE
gfal-copy file://$(pwd)/MyAwesomeFile gsiftp://se01.dur.scotgrid.ac.uk/dpm/dur.scotgrid.ac.uk/home/pheno/MyAwesomeFile
Remove a File or Directory
To Remove things; Just like rm you need to use -r for directories, there is also a –dry-run option to check what it will do before hand.
Stream File to StdOut
To stream a file via stdout from the SE (Useful for file chunking)
gfal-cat <protocol>://<se_address>/dpm/<site_domain>/home/<vo>/<username>/<filename> > ~/<filename>
Stream StdIn to File
To stream from stdin to the SE (Useful for outputs)
cat /etc/os-release | gfal-save <protocol>://<se_address>/dpm/<site_domain>/home/<vo>/<username>/os-release
LFC – Logical File Catalog
This system is being phased out due to lack of official support and we can no longer support it. It is only mentioned here for documentation reasons.
Please use the gfal suite and target the LFC as well as the gfal-legacy-* commands.
CVMFS – Cern Virtual Machine File System
This is a read only HTTP file system that you can interact with as though it’s a standard local file system, it is available across all GridUIs and nodes that support GridPP work, if a site supports Pheno it should have the Pheno directory available. Each VO have their own directories.
Pheno can be found in /cvmfs/pheno.egi.eu
If you want write access to this then please contact us as it is restricted access and a complex method and we don’t explain it on a public guide.