November 18, 2023
Overview of Cluster Attached Storage on Scientific Computing (SciC) Linux Clusters (CASE)
Table of Contents
BriefCASE storage
Please note that while snapshots of changed files are kept for a short time, SciC Linux Clusters storage is not backed-up. Please back up your essential data.
You can check your directory usage with the command
/usr/local/bin/pan_quota $HOME
Home Folders under /PHShome
Every account on the SciC Linux Clusters includes a home directory folder with a soft quota of 160GB and an upper limit of 400GB (quotas correct on 1/22/2018).
Shared group folders under /data
The BriefCASE storage option is high-performance storage attached to the cluster ONLY for data used for analysis on SciC Linux Clusters. It supports large-scale parallel processing of genomic and medical imaging data sets. BriefCASE storage is issued to Lab groups as a shared resource per Lab group or per project and can scale from 200GB to 50+ TB. This is charged monthly for space used at approximately $17 per TB per month (see our Core Services Catalog for up-to-date information).
(Data protection is provided by daily snapshots that retain deleted or changed files for 2-7 days depending on recent activity. *2023/11/17 Snapshot is currently suspended owing to storage constraints.*)
Request BriefCASE storage at https://rcservices.partners.org/
Working with temp/scratch files
Introduction
Many applications create temporary files as they run, which are deleted after the final output is saved. Typically the default behavior is for these files to be created in the "current working directory" or the /tmp folder, but the location where they are created can usually be set to a location of your choice.
On the SciC Linux Clusters you may not use /tmp as a location for temporary files, because doing so can cause /tmp to run out of space and your job or jobs belonging to other investigators to fail.
/scratch or /data folder
If your lab group has a /scratch or /data folder, these are the best locations for temporary files. Create your own temp folder inside the /scratch or /data folder and set the application options to write to this folder. Consult the application manual for how to set this option ( it may be an option like --tmp )
Personal scratch folder
Each cluster account comes with a scratch folder for temporary files at $HOME/scratch which can store up to 600GB for a short period. Files in personal scratch folders are deleted automatically after 20 days, so this storage space is only suitable for transient files, and tasks that are expected to run for a week or less.
The location "$HOME/scratch" is a link (shortcut) to your scratch location. If yours is not present, recreate the link with the commands
cd $HOME ; ln -s /scratch/${USER:0:1}/$USER scratchCheck how much remains in your scratch quota with the commands
cd $HOME/scratch; /usr/local/bin/pan_quota The columns <GB> <soft> <hard> refer to current usage and the soft and hard quota for disk space while the columns <files> <soft> <hard> refer to the current number of files and the soft and hard quota for number of files. If either hard quota is reached, no more files can be written to scratch.
Reserving space under /tmp through the LSF scheduler
Use this option only at the direction of the Scientific Computing support team. Space in the /tmp folder on compute nodes is a resource that can be reserved through the job scheduler. The scheduler provides a dedicated temp file location under /tmp for each job that is run, passing the location in an environment variable "$TMPDIR". When the job completes this temp folder will be removed even if the job exits prematurely which helps prevent /tmp filling up. To use this option, set the option of your application to write temporary files to "${TMPDIR}" and also add a scheduler option to reserve the amount of disk space that the job will use under /tmp, for example to reserve 5GB of space, use
-R "rusage[tmp=1000]" The units are MB
Other Storage Categories
These FAQ pages discuss additional storage options not covered here:
A common "scratch" space is planned but not currently available
Redundancy levels
Traditional RAID protection distributes the file and parity-check data across independent disk drives. BriefCASE instead distributes file and parity data across independent file servers, and this is termed "object-RAID".
BriefCASE redundancy levels can be set on a per-file or per-directory basis. By default files under 64kB are stored with object-RAID1 and files over 64kB are stored with object-RAID5. In both cases, a second orthogonal layer of protection is applied, termed "vertical parity". Object-RAID5 + vertical parity provides a comparable level of protection to traditional RAID6, with the advantage that object-RAID rebuild times are much shorter. Object-RAID6 will be available in a future release. Vertical parity overhead is 3% (1 in 32 sectors), and it protects against loss of a disk sector.
The redundancy overhead of Object-RAID5 + vertical parity for large files is 1.143570, meaning that 1GB of data occupies 1.143570GB of formatted disk space. The "du" (disk usage) linux command by itself gives the size-on-disk, and with the "--apparent-size" flag gives the actual file size:
du -B 1000000 test.bam
195764 .test.bam
du --apparent-size -B 1000000 test.bam
171208 test.bam
Folder usage
BriefCASE includes utilities optimized for to that filesystem. "pan_du" calculates disk usage based on file size and number of files in each folder. Note that the default 20 parallel threads places a heavy load on the system and should not be used except off-hours. Instead use
pan_du -t 2 /data/lab_folderA usage report is automatically generated each week for /data folders on BriefCASE. Look for a file named "usage_report.txt". To quickly identify files and folders taking up the most space, use the command
sort -n -k 5 usage_report.txt | tail -n 100Quotas
Each folder has an associated hard and soft quota. The hard quota is a fixed limit is the value in the "Size" column seen with the "df" command:
cd /data/test
df -H .
Filesystem Size Used Avail Use% Mounted on
panfs://10.129.86.180/hpc/groups/test
64T 49T 16T 77% .
The "Used" column reported by "df" does not match the billed usage so don't rely on this value. It may include the amount of space occupied by snapshots (we're not sure).
Email Scientific Computing to request a quota expansion if you anticipate exceeding the current quota. There is also a soft quota set at a lower value which alerts ERIS HPC staff that a folder is close to quota, you can request to be included on this email alert.
Extended Attributes
Extended attributes can be used to set the redundancy level, file locking settings and extended permissions for a file or folder