Locked History Actions

Diff for "Computing/LIP_Lisbon_Farm/3_Data_Management"

Differences between revisions 29 and 30
⇤ ← Revision 29 as of 2017-12-12 17:21:48 →
Size: 4800
Editor: lalves
Comment:
← Revision 30 as of 2017-12-12 17:24:48 → ⇥
Size: 4792
Editor: lalves
Comment:
Deletions are marked like this. Additions are marked like this.
Line 64: Line 64:
=== Install sshfs package === == Install sshfs package ==
Line 77: Line 77:
=== Create a local directory === == Create a local directory ==
Line 82: Line 82:
=== Mount your remote folder === == Mount your remote folder ==
Line 87: Line 87:
=== When you're done, you can unmount it === == When you're done, you can unmount it ==

Contents

  1. File systems available
  2. Access Policy
  3. Directories and filesystems
  4. Important Disclaimer
  5. Access your data by SSHFS
    1. Install sshfs package
    2. Create a local directory
    3. Mount your remote folder
    4. When you're done, you can unmount it

File systems available

  • The users have at there disposal two types of filesystem:

    • A common NFS filesystem for homes (20 Gb per user) and the exper-sw where users can include there own software. Each individual group at LIP as there own software area (exper-sw) but access it's restricted.

      • Individual Write access to /exper-sw/ area it's granted by the group leader.

    • A lustre filesystem for each individual group. Lustre is a shared filesystem (present in the execution hosts and in the submission hosts) dedicated for the storage of big/huge files

  • Groups involved in WLCG transfer data using grid technologies can also use the Tier-3 storage areas and dedicated software areas (CVMFS) made available by there communities.

Access Policy

  • By default both lustre and nfs individual group areas can be accessed (read only) by users of the same group.

Directories and filesystems

  • The fermi machines provide a wide set of filesystems so that users can access their data and applications.
  • The available filesystems in login machines are: 

# df -h
Filesystem              Size  Used Avail Use% Mounted on
nfs01:/exports/exper-sw 2.8T  2.0T 767G  73% /exper-sw                  ---> NFS fs for local software
se16:/ams             200G   33M   200G   1% /home/ams                  ---> NFS fs LIP homes and data
se16:/comp            600G  6.8G   594G   2% /home/comp                 ---> NFS fs LIP homes and data
se16:/cosmo           600G  8.8G   591G   2% /home/cosmo                ---> NFS fs LIP homes and data
se16:/feno            200G   33M   200G   1% /home/feno                 ---> NFS fs LIP homes and data
se16:/nucria          600G   33M   600G   1% /home/nucria               ---> NFS fs LIP homes and data
se16:/pet             200G   33M   200G   1% /home/pet                  ---> NFS fs LIP homes and data
se16:/sno             200G  8.2G   192G   5% /home/sno                  ---> NFS fs LIP homes and data
se16:/t3atlas         200G  151G   49G   76% /home/t3atlas              ---> NFS fs LIP homes and data
se16:/t3cms           200G   29G   172G  15% /home/t3cms                ---> NFS fs LIP homes and data
se17:/x               4.6T  4.4T   165G  97% /x                         ---> NFS fs LIP ATLAS users
se27:/ams              11T  5.9T   4.2T  59% /z/ams                     ---> NFS    FS for AMS   users
se27:/comp             11T  3.8T   6.3T  38% /z/comp                    ---> NFS    FS for COMP  users
se16:/csys            200G   33M   200G   1% /home/csys                 ---> Lustre FS for AUGER users
mdt02@tcp:/t3atlas     61T   24T   34T   42% /gstore/t3atlas            ---> Lustre FS  Tier-3 LIP ATLAS users 
mdt02@tcp:/T3CMS       81T   11T   66T   15% /gstore/t3cms              ---> Lustre FS  Tier-3 LIP CMS   users
mdt03@tcp:/calo        72T   62T   6.6T  91% /lstore/calo               ---> Lustre FS for ATLAS users
mdt02@tcp:/cmslocal   6.4T  5.6T   541G  92% /lstore/cms                ---> Lustre FS for CMS   users
mdt03@tcp:/comp        41T  9.0T   30T   24% /lstore/comp               ---> Lustre FS for COMP  users
mdt03@tcp:/sno         10T  6.4T   3.2T  67% /lstore/sno                ---> Lustre FS for SNOW  users
mdt04@tcp:/auger       36T   22T   12T   65% /lstore/auger              ---> Lustre FS for AUGER users
mdt04@tcp:/hpclip     6.4T  546M   6.0T   1% /lstore/hpclip             ---> Lustre FS for HPC   users
mdt04@tcp:/lattes     5.9T   86G   5.5T   2% /lstore/lattes             ---> Lustre FS for LATTS users
mdt04@tcp:/pet        461G  115G   323G  27% /lstore/pet                ---> Lustre FS for PET   users
cvmfs2                 20G   17G   3.3G  84% /cvmfs/cms.cern.ch         ---> CVMFS  FS for CMS   users
cvmfs2                 20G  6.1G   14G   32% /cvmfs/atlas.cern.ch       ---> CVMFS  FS for ATLAS users

Important Disclaimer

  • Manipulating huge sets of small files generates performance degradation issues in /lustre due to the lock management. Therefore, you should not

    • Compile anything under /lustre

    • Store and access databases under /lustre

Access your data by SSHFS

Install sshfs package

On Ubuntu/debian: 

# sudo apt-get install sshfs

On RHEL/CentOS/Fedora: 

# sudo yum install fuse-sshfs

Create a local directory

# mkdir ~/homeNCG

Mount your remote folder

# sshfs $USER@fermi.ncg.ingrid.pt:/home/$GROUP/$USER ~/homeNCG

When you're done, you can unmount it

# fusermount -u ~/homeNCG
last edited 2024-03-07 11:37:09 by martinsj