Storage – Grid

To use the grid storage you must first have a grid certificate.

Various storage systems are available which should enable you to run your jobs and efficiently scale them. It is recomended you store your files between each gateway if you require long term storage and/or the data is difficult to replicate. Grid Storage is not automatically backed up!

Xroot Storage Gateway – Durham

Xroot Storage Gateway – Glasgow

The Storage Gateway at Durham is an object file system that utilises Xrootd and Ceph. There are several protocols available which are listed above. We highly recommend that people utilise the Gateway for all Jobs that exceeds 10MB in either input or output files or both combined.

You will find that although this is an extra layer of complexity, the speed, capacity and robustness is worth the effort. Xroot/Ceph is constantly developing and changing so if you’re struggling get in touch.

Important – Permission Differences

The permissions for the Pheno is group based storage has changed; all pheno users have full access rights to any other pheno user data, this means you can write/edit/read data from user users so please proceed with caution. There are future plans will permit more granular permissions but this is currently not available to us.

GFAL2 Toolset

The gfal2 toolset should feel relatively familiar to the standard Linux file tools. Be aware that if you use the file:// handler for local files, you need to include the full directory path. Also be aware that not all sites conform to the same standard.

Make a Directory

To make a directory

gfal-mkdir <protocol>://<se_address>/gridstore/<site_domain>/home/<vo>/<username>

For example

gfal-mkdir xroot://grid-xgw-01.dur.scotgrid.ac.uk:1094/gridstore/dur.scotgrid.ac.uk/home/pheno/<username>

View a Directory

To view a directory

gfal-ls <protocol>://<se_address>/gridstore/<site_domain>/home/<vo>/<username>

For example

gfal-ls xroot://grid-xgw-01.dur.scotgrid.ac.uk:1094/gridstore/dur.scotgrid.ac.uk/home/pheno/MyUserName

Copy a File

To copy a file  ( -f flag may be added to override existing files, CAUTION)

gfal-copy <protocol>://<file_source> <protocol>://<file_source>

For example to copy a file from the Gateway

gfal-copy xroot://grid-xgw-01.dur.scotgrid.ac.uk:1094/gridstore/dur.scotgrid.ac.uk/home/pheno/MyUserName/MyAwesomeFile file://$(pwd)/MyAwesomeFile

For example to copy a file to the SE

gfal-copy file://$(pwd)/MyAwesomeFile xroot://grid-xgw-01.dur.scotgrid.ac.uk:1094/gridstore/dur.scotgrid.ac.uk/home/pheno/MyAwesomeFile

Remove a File or Directory

To Remove things; Just like rm you need to use -r for directories, there is also a –dry-run option to check what it will do before hand.  

gfal-rm <protocol>://<se_address>/dpm/<site_domain>/home/<vo>/<username>/<file_or_dir>

For example

gfal-rm xroot://grid-xgw-01.dur.scotgrid.ac.uk:1094/gridstore/dur.scotgrid.ac.uk/home/pheno/MyUserName/MyAwesomeFile

Stream File to StdOut

To stream a file via stdout from the SE (Useful for file chunking)

gfal-cat <protocol>://<se_address>/gridstore/<site_domain>/home/<vo>/<username>/<filename> > ~/<filename>

Stream StdIn to File

To stream from stdin to the SE (Useful for outputs)

cat /etc/os-release | gfal-save <protocol>://<se_address>/gridstore/<site_domain>/home/<vo>/<username>/os-release

Old SE01 aka DPM

This system is now retired. If you require any data from it, please get in touch before the end of 2023.

CVMFS – Cern Virtual Machine File System

This is a read only HTTP file system that you can interact with as though it’s a standard local file system, it is available across all GridUIs and nodes that support GridPP work, if a site supports Pheno it should have the Pheno directory available. Each VO have their own directories.

Pheno can be found in /cvmfs/pheno.egi.eu

If you want write access to this then please contact us as it is restricted access and a complex method and we don’t explain it on a public guide.