Locked History Actions

Diff for "Computing/LIP_Lisbon_Farm/3_Data_Management"

Differences between revisions 1 and 33 (spanning 32 versions)
⇤ ← Revision 1 as of 2016-12-07 14:45:30 →
Size: 2524
Editor: jpina
Comment:
← Revision 33 as of 2021-02-19 19:38:08 → ⇥
Size: 5152
Editor: martinsj
Comment:
Deletions are marked like this. Additions are marked like this.
Line 1: Line 1:
== Directories and filesystems ==
 * The ui6 machines provide a wide set of filesystems so that users can access their data and applications.		 ## page was renamed from Computing/LIP_Lisbon_Farm/Data_Management
## page was renamed from Computing/LIP_Farm/Data_Management
<<TableOfContents()>>
Line 4: Line 5:
 * The available filesystems in LIP-Lisbon login machines are: = File systems available =

 * The users have at there disposal three types of filesystem:
  * A '''common NFS filesystem''' for homes not shared with workernodes
  * A '''lustre filesystem''' for some groups shared filesystem with the workernodes
  * A '''CVMFS read-only filesystem''' for software provision
 * Groups involved in WLCG transfer data using grid technologies can also use the Tier-3 storage areas and dedicated software areas (CVMFS) made available by there communities.

= Access Policy =

 * By default both lustre and nfs individual group areas can be accessed (read only) by users of the same group.


= Directories and filesystems =

The '''pauli''' machines provide a wide set of filesystems so that users can access their data and applications.

== NFS Home directories ==

Users home mounts are not shared with workernodes, they can be found only on submission nodes:
Line 7: Line 27:
# df -h
Filesystem Size Used Avail Use% Mounted on
mdt02@tcp:/atlas 12T 8.4T 2.7T 76% /lustre/atlas ---> Lustre FS for ATLAS Tier-2 and Tier-3 grid activities
mdt02@tcp:/auger 28T 24T 2.9T 90% /lustre/auger ---> Lustre FS for AUGER local users
mdt02@tcp:/calor 72T 37T 32T 54% /lustre/calo ---> Lustre FS for CALO local users
mdt02@tcp:/pem 186G 118G 60G 67% /lustre/pet ---> Lustre FS for PET local users
mdt02@tcp:/sno 10T 1.1T 8.5T 12% /lustre/sno ---> Lustre FS for SNO local users
st011:/exports/soft 20G 12G 7.3G 61% /soft ---> NFS fs for local software
st011:/exports/lip-tmp 2.0T 126G 1.9T 7% /hometmp ---> NFS fs for temporary / scratch storage
st011:/exports/home 20G 12G 7.3G 61% /v/home ---> NFS fs LIP homes and data
st002:/exports/home 22G 6.2G 14G 31% /u/home ---> NFS fs LIP homes and data
st002:/exports/data 22G 6.2G 14G 31% /u/data ---> NFS fs LIP homes and data
st002:/exports/x 12T 9.7T 2.2T 82% /x ---> NFS fs LIP homes and data
st012:/exports 20G 3.8G 15G 20% /z ---> NFS fs LIP homes and data		 FILESYSTEM MOUNTED ON SCOPE
se16:/ams /home/ams AMS
se16:/comp /home/comp COMPASS
se16:/cosmo /home/cosmo AUGER
se16:/csys /home/csys CSYS
se16:/feno /home/feno FENO
se16:/nucria /home/nucria NUCRIA
se16:/pet /home/pet PET
se16:/sno /home/sno SNO PLUS
se16:/t3atlas /home/t3atlas ATLAS
se16:/t3cms /home/t3cms CMS
Line 22: Line 39:
 * The available filesystems in LIP-Coimbra login machines are:
== NFS Data directories ==

Projects specific data NFS mounts are not shared with workernodes and are found only on submission nodes:
Line 25: Line 45:
$ df -h
Filesystem Size Used Avail Use% Mounted on
lustre@tcp:/gstore 101T 71T 25T 75% /gstore ---> Lustre fs dedicated for atlas tier2 grid activities
llustre@tcp:/lstore-1 34T 9.4T 23T 30% /lstore/atlaslocalgroupdisk ---> Lustre fs dedicated for atlas tier3 grid users
llustre@tcp:/lstore-2 43T 38T 2.9T 93% /lstore/atlas ---> Lustre fs dedicated for atlas local users
llustre@tcp:/lstore-3 18T 12T 5.5T 68% /lstore/lip ---> Lustre fs dedicated for lip local users
192.168.2.44:/software 3.9T 882G 3.0T 23% /software ---> NFS fs for local software
192.168.2.30:/exports/home-atlas 13T 12T 1.5T 89% /home/local/atlas ---> NFS fs for ATLAS local homes
192.168.2.30:/exports/home-lip 13T 12T 1.4T 90% /home/local/lip ---> NFS fs for LIP local homes		 se17:/x /x ATLAS data
se27:/ams /z/ams AMS data
se27:/comp /z/comp COMPASS data
Line 35: Line 49:

== LUSTRE Data directories ==

Projects specific data LUSTRE mounts are shared with workernodes:

{{{
mdt02@tcp:/t3atlas /gstore/t3atlas ATLAS Tier-3
mdt02@tcp:/T3CMS /gstore/t3cms CMS Tier-3
mdt04@tcp:/auger /lstore/auger AUGER
mdt10@tcp:/atlasloc /lstore/atlas ATLAS Coimbra
mdt04@tcp:/calo /lstore/calo ATLAS Lisboa
mdt02@tcp:/cmslocal /lstore/cms CMS
mdt03@tcp:/comp /lstore/comp COMP
mdt04@tcp:/lattes /lstore/lattes LATTES
mdt04@tcp:/nucria /lstore/nucria NUCRIA
mdt04@tcp:/pet /lstore/pet PET
mdt10@tcp:/sno /lstore/sno SNO PLUS
}}}

== CVMFS Software provision ==

The '''CVMFS''' is a read-only filesystem with cache local to clients optimal for software provision:

{{{
cvmfs2 /cvmfs/cms.cern.ch CMS VO
cvmfs2 /cvmfs/atlas.cern.ch ATLAS VO
cvmfs2 /cvmfs/sw.el7 LIP
}}}


= Important Disclaimer =
 * Manipulating huge sets of small files generates performance degradation issues in /lustre due to the lock management. Therefore, you should not
  * Compile anything under '''/lustre'''
  * Store and access databases under '''/lustre'''

= Access your data by SSHFS =

== Install sshfs package ==

On Ubuntu/debian:
{{{
# sudo apt-get install sshfs
}}}

On RHEL/CentOS/Fedora:
{{{
# sudo yum install fuse-sshfs
}}}


== Create a local directory ==
{{{
# mkdir ~/homeNCG
}}}

== Mount your remote folder ==
{{{
# sshfs martinsj@fermi.ncg.ingrid.pt: ~/homeNCG
}}}

== When you're done, you can unmount it ==
{{{
# fusermount -u ~/homeNCG
}}}

== .bashrc tunning ==

In case you're dealing with very long paths, as a suggestion, you can add to your ~/.bashrc file the following:

{{{
alias mfermi='f(){ dir=$1 ; [ -n "$dir" ] || dir=homeNCG ; [ -d ~/$dir ] || mkdir -p ~/$dir ; sshfs fermi.ncg.ingrid.pt: ~/$dir ; unset -f f; }; f'
alias ufermi='f(){ dir=$1 ; [ -n "$dir" ] || dir=homeNCG ; [ -d ~/$dir ] && fusermount -u ~/$dir ; [ -d ~/$dir ] && rmdir ~/$dir ; unset -f f; }; f'

alias mncg='f(){ dir=$1 ; [ -n "$dir" ] && ldir=`basename $dir` ; [ -d ~/$ldir ] || mkdir -p ~/$ldir ; [ -n "$dir$ldir" ] && sshfs fermi.ncg.ingrid.pt:$dir ~/$ldir ; unset -f f; }; f'
alias uncg='f(){ dir=$1 ; [ -n "$dir" ] && ldir=`basename $dir` ; ldir=`echo $ldir ; [ -d ~/$ldir ] && fusermount -u ~/$ldir ; [ -d ~/$ldir ] && rmdir ~/$ldir ; unset -f f; }; f'

alias mcalo='f(){ [ -d lcalo ] || mkdir ~/lcalo ; sshfs fermi.ncg.ingrid.pt:/lstore/calo ~/lcalo ; unset -f f; }; f'
alias ucalo='f(){ [ -d ~/lcalo ] && fusermount -u ~/lcalo ; [ -d ~/$ldir ] && rmdir ~/lcalo ; unset -f f; }; f'
}}}

This is how you can use it:

{{{
$ mfermi homeNCG
$ df
Filesystem 1K-blocks Used Available Use% Mounted on
fermi.ncg.ingrid.pt: 209612800 3504112 206108688 2%
/home/martinsj/homeNCG
$ ufermi homeNCG

$ mncg /lstore/calo
...
$ uncg /lstore/calo

}}}

Contents

  1. File systems available
  2. Access Policy
  3. Directories and filesystems
    1. NFS Home directories
    2. NFS Data directories
    3. LUSTRE Data directories
    4. CVMFS Software provision
  4. Important Disclaimer
  5. Access your data by SSHFS
    1. Install sshfs package
    2. Create a local directory
    3. Mount your remote folder
    4. When you're done, you can unmount it
    5. .bashrc tunning

File systems available

  • The users have at there disposal three types of filesystem:

    • A common NFS filesystem for homes not shared with workernodes

    • A lustre filesystem for some groups shared filesystem with the workernodes

    • A CVMFS read-only filesystem for software provision

  • Groups involved in WLCG transfer data using grid technologies can also use the Tier-3 storage areas and dedicated software areas (CVMFS) made available by there communities.

Access Policy

  • By default both lustre and nfs individual group areas can be accessed (read only) by users of the same group.

Directories and filesystems

The pauli machines provide a wide set of filesystems so that users can access their data and applications.

NFS Home directories

Users home mounts are not shared with workernodes, they can be found only on submission nodes: 

FILESYSTEM            MOUNTED ON                 SCOPE
se16:/ams             /home/ams                  AMS
se16:/comp            /home/comp                 COMPASS
se16:/cosmo           /home/cosmo                AUGER
se16:/csys            /home/csys                 CSYS
se16:/feno            /home/feno                 FENO
se16:/nucria          /home/nucria               NUCRIA
se16:/pet             /home/pet                  PET
se16:/sno             /home/sno                  SNO PLUS
se16:/t3atlas         /home/t3atlas              ATLAS
se16:/t3cms           /home/t3cms                CMS

NFS Data directories

Projects specific data NFS mounts are not shared with workernodes and are found only on submission nodes: 

se17:/x               /x                         ATLAS data
se27:/ams             /z/ams                     AMS data
se27:/comp            /z/comp                    COMPASS data

LUSTRE Data directories

Projects specific data LUSTRE mounts are shared with workernodes: 

mdt02@tcp:/t3atlas    /gstore/t3atlas            ATLAS Tier-3
mdt02@tcp:/T3CMS      /gstore/t3cms              CMS Tier-3
mdt04@tcp:/auger      /lstore/auger              AUGER
mdt10@tcp:/atlasloc   /lstore/atlas              ATLAS Coimbra
mdt04@tcp:/calo       /lstore/calo               ATLAS Lisboa
mdt02@tcp:/cmslocal   /lstore/cms                CMS
mdt03@tcp:/comp       /lstore/comp               COMP
mdt04@tcp:/lattes     /lstore/lattes             LATTES
mdt04@tcp:/nucria     /lstore/nucria             NUCRIA
mdt04@tcp:/pet        /lstore/pet                PET
mdt10@tcp:/sno        /lstore/sno                SNO PLUS

CVMFS Software provision

The CVMFS is a read-only filesystem with cache local to clients optimal for software provision: 

cvmfs2                /cvmfs/cms.cern.ch         CMS VO
cvmfs2                /cvmfs/atlas.cern.ch       ATLAS VO
cvmfs2                /cvmfs/sw.el7              LIP

Important Disclaimer

  • Manipulating huge sets of small files generates performance degradation issues in /lustre due to the lock management. Therefore, you should not

    • Compile anything under /lustre

    • Store and access databases under /lustre

Access your data by SSHFS

Install sshfs package

On Ubuntu/debian: 

# sudo apt-get install sshfs

On RHEL/CentOS/Fedora: 

# sudo yum install fuse-sshfs

Create a local directory

# mkdir ~/homeNCG

Mount your remote folder

# sshfs martinsj@fermi.ncg.ingrid.pt: ~/homeNCG

When you're done, you can unmount it

# fusermount -u ~/homeNCG

.bashrc tunning

In case you're dealing with very long paths, as a suggestion, you can add to your ~/.bashrc file the following: 

alias mfermi='f(){ dir=$1 ; [ -n "$dir" ] || dir=homeNCG ; [ -d ~/$dir ] || mkdir -p ~/$dir ; sshfs fermi.ncg.ingrid.pt: ~/$dir ;  unset -f f; }; f'
alias ufermi='f(){ dir=$1 ; [ -n "$dir" ] || dir=homeNCG ; [ -d ~/$dir ] && fusermount -u ~/$dir ; [ -d ~/$dir ] && rmdir ~/$dir ;  unset -f f; }; f'

alias mncg='f(){ dir=$1 ; [ -n "$dir" ] && ldir=`basename $dir` ; [ -d ~/$ldir ] || mkdir -p ~/$ldir ; [ -n "$dir$ldir" ] && sshfs fermi.ncg.ingrid.pt:$dir ~/$ldir ;  unset -f f; }; f'
alias uncg='f(){ dir=$1 ; [ -n "$dir" ] && ldir=`basename $dir` ; ldir=`echo $ldir ; [ -d ~/$ldir ] && fusermount -u ~/$ldir ; [ -d ~/$ldir ] && rmdir ~/$ldir ;  unset -f f; }; f'

alias mcalo='f(){ [ -d lcalo ] || mkdir ~/lcalo ; sshfs fermi.ncg.ingrid.pt:/lstore/calo ~/lcalo ;  unset -f f; }; f'
alias ucalo='f(){ [ -d ~/lcalo ] && fusermount -u ~/lcalo ; [ -d ~/$ldir ] && rmdir ~/lcalo ;  unset -f f; }; f'

This is how you can use it: 

$ mfermi homeNCG
$ df
Filesystem           1K-blocks    Used Available Use% Mounted on
fermi.ncg.ingrid.pt: 209612800 3504112 206108688   2%
/home/martinsj/homeNCG
$ ufermi homeNCG

$ mncg /lstore/calo
...
$ uncg /lstore/calo
last edited 2024-03-07 11:37:09 by martinsj