SAM-FS software provides services which automatically manage the migration and retrieval of files between multiple levels of a storage hierarchy: from on-line disk cache, to nearline tape archival, to offline tape archival.

SAM-FS software allows the user to copy files onto nearline media immediately after creation - so that data on disk doesn't remain at risk until so called "high water marks" are reached. Up to four copies of data can be made to different types of media (at the APAC National Facility we use two) in order to protect against media malfunction and create media volume sets for off-site storage. Files are copied to removable media in tar format and file metadata is copied to tape for easy retrieval. Archive files or complete file systems can be moved or duplicated on multiple SAM-FS servers, or off-site servers, for additional data protection.

SAM-FS archived files are written in tar format so that data can be easily be recovered in minutes on any UNIX system (with or without SAM-FS software). Removable volumes (nearline and off-line) have ANSI-standard labels.

 
WARNING

Operational considerations prevent us currently from keeping off-site copies of MDSS data for most projects. While two copies are kept on separate tapes in the tape silo, we cannot recover such data in the event of a catastrophic event destroying both those copies. The National Facility can allocate the second copy of data to a smaller silo at a second location at the ANU that is used for some projects with non-recreatable data. Contact help@nf.apac.edu.au if you wish your project to be considered. See here for more details.

SAM-FS migration is generally transparent to users. A short time after a file is written to (or modified on) the disk cache, the file will automatically be written to archival tape. The usual Unix file management commands all work - 'cp', 'rm', 'mkdir', 'vi', etc. However to achieve the best throughput, see the following section on Planning for Efficient Use.

SAM-FS file retrieval from tape is characterized by a delay while the tape is loaded onto the tape drive and the file is copied to cache. This layup time depends on both drive availability (due to high demand) and the speed of the Tape robot mechanism. System performance is monitored and refined in order to provide optimal tape request turnaround.

Users accessing files via an NFS mount on their local system may see frequent "NFS not responding. Still trying" messages while the file is being staged back onto the disk cache.

Since data retrieval from a tape has a latency due to tape mounts etc there are a number of special tools available on the MDSS host, store, to determine the physical location of files, and for enabling bulk file stagein and stageout. To use the tools requires logging onto the Mass Data Storage system, or by using special scripts designed by National Facility staff. If your project does have login access to store then you can try these commands yourself. Getting to know these tools is not necessary for many operations, but is a must for those projects requiring optimal performance. Please contact our consultants at help@nf.apac.edu.au for assistance in using these features.

The host store is managed using the Solaris UNIX operating system with the following additional SAM-FS file system tools for file migration. Refer to the store online man pages for more information.

• archive - archive files to tapes
• stage - retrieve files from the tape to disk cache
• sls - similar to ls, with more file migration information
• sfind - SAM-FS find
• release - release the file from disk cache
• ssum - set file checksum attributes
• sdu - du replacement size of archived directory/file.

Projects dealing with massive amounts of data need to carefully consider all aspects of data acquisition, storage, retrieval, navigation, and interpretation. Refer to the section: Consulting Support for assistance.

Projects are advised to carefully consider the data flow requirements of their data retrieval and navigation since this facet of MDSS will be the most utilized in the long term.

Following are some issues to be considered prior to writing the first byte of data to SAM-FS:

1. data organization;
2. directory structure for efficient recall;
3. file layout for efficient recall;
4. data navigation (aka metadata) database for archived directories and files;
5. projected access/retrieval modes;
6. user and application interface to data retrieval process.

Archival of small files ( < 20Mb) places a processing burden on SAM-FS and hence reduces overall throughput to all users. Users with many small files are requested to bundle them into larger files. Common tools used to bundle files are tar(1) and cpio(1). Although it's preferable to bundle files on your local machine we have also a short term area on host store that can be used to bundle files before being saved in a SAM-FS filesystem.

Example:

# transfer files to store from an sc node (perhaps as part of a job script)

scnode> tar cf archivename.tar filelist
scnode> cp archivename.tar /massdata/projectcode

# If your project has an account on store (contact us if you believe you need access) then bundle files on remote system, then archive

Local> tar cf archivename.tar filelist
Local> ftp store
binary
put archivename.tar
quit

# bundle files then ship container file to STORE.

Local> tar cfb - 20 filelist | rsh store dd of=archive.tar obs=20b

A variety of status information is available for each SAM-FS file. Read the man pages for sls(1) for information on displaying file attributes. Read sfind(1) for information on searching a directory for files having specified attributes.

Example:

sls -2 file1 file2

sls -D file1 file2

sfind dirname -archived

Prestaging files which will be used in the near future ensures there is no delay when either a user, during an interactive session or a batch code, accesses them. Two methods are available: explicit staging or associative staging.

Explicit staging transfers one or more files from archive tape to the disk cache. The default command returns immediately after marking the files for asynchronous stage-in. In order to force the files to be staged prior to the command's completion, use the -w option. Refer to stage(1) for more details.

Example:

# prestage 2 (non-associative) archived files from tape to disk

stage -w file1 file2

# Recursively prestage all files in named subdirectory

stage -r -w dirname

Note that when staging many files at once (such as with stage -r -w) the "-w" option causes each file to become completely staged before the stage request for the next file is issued. This does not allow the system to sort the stage requests in the order that the files are archived on the media. In order to get the best performance in this situation, do the following:

stage -r dirname ; stage -r -w dirname

The associative staging attribute, set by the user on a directory, causes the automatic staging onto disk cache of all files within the directory when any one of the associated files is accessed.

Example:

# set attribute for associate staging on on files in a single directory
# The parent directory is also marked

stage -a filename ... filename

# set attribute for associate staging on a single directory

stage -a dirname

# set attribute for associate staging on all subdirectories

stage -r -a dirname

# prestage all files associated with file1

stage file1

Following are SAM-FS specific attributes which the user may set. Refer to release(1), archive(1), and stage(1) for details.

Archive attributes:

No Archive (-n) File will not be archived.

Release attributes:

Partial (-p) First portion of file will be retained in disk cache.

After archive (-a) File will be released from disk cache after archive.

Stage attributes:

No stage (-n) File will not be staged; data will be read directly from the archive media.

Associative (-a) File is part of an associative stage group.

The Mass Data Store is connected to the APAC peak facility using gigabit links to enable high bandwidth data transfer capability. The GrangeNet broadband network is used to connect to institutions and centres that have signed up as partners, using multiple gigabit links. This allows fast storage and retrieval of data from the APAC partners throughout Australia using more distributed types of projects. The status of the GrangeNet connections is monitored via the GrangeNet weather station. The store is also connected to the local area network node ( CARNO) of the Australian Research Network (AARNet), and the ANU Internet backbone using gigabit links.

The order of preference for transfer commands is:

1. netmv/netcp
2. scp
3. ftp

A temporary storage area /short can be requested for users' needs to save large files for a short time. It will be serviced by a filesystem of 130 Gbytes, with limits If you would like to make use of this storage or if you would like more information about this service, please send an email to help@nf.apac.edu.au

 
Numerous Small Files

Maintaining user directories containing many small files places a great burden on SAM-FS. The inherent overhead of archival processing for each file is large.

To quantify 'many' and 'small': an account, having more than 30% of its files smaller than 10 MB, reaches our cautionary limit. When this threshold is reached, the user is required to commence bundling the small files into larger container files (e.g. using 'tar' or 'cpio').

We have a local short term area to assist in bundling smaller files.

 
Backups of Quickly Changing Source Code

Archive copies of source trees are a common use of archival systems. The preferred backup model is to manage the source under a configuration management tool and then backup only the major software milestones. The configuration manager will manage the day-to-day changes on the user's local disk.

The APAC National Facility consultants will consult on the selection, integration and use of common public domain configuration management tools (e.g. CVS, SCCS, RCS). Providing methods of tracking daily changes (thus enabling easy backout of injudicious modifications) and major branches in code development.

 
Locking Files onto Disk Cache

Locking the initial kilobytes of a file onto disk cache will speed the initial flow of data to the requester. However, the disk cache could easily become full with locked down data if users routinely set this attribute. A small number of such locked files is permissible. APAC National Facility staff will closely monitor the use of this attribute for abuse. If this attribute is required by your project, you need to write to help@nf.apac.edu.au.

To prevent user login delay, the standard startup files will be automatically, in their entirety, locked onto the disk when they are found during a weekly automated search. The default locked files are: .cshrc, .profile, .login, .rhosts, .logout, .history, and .sshrc/*.

In order to prevent the MDSS decaying into a repository for WORN (write-once, read-never) data, the APAC National Facility monitors each project's usage. Projects demonstrating inappropriate usage patterns may be requested to move their data to off-line storage (if data is never accessed), to on-line local storage (if excessive real-time use), or to change their storage model (if too many small files).

Recovery of files inadvertently removed or mangled is possible within the period of a week. The short timeframe is due to the possibility that the tape holding the original file might be returned to the tape pool.

The recovery process is complex and not automatic. SAM-FS does not version pre-existing files of the same name--so the recovery involves reloading an old snapshot of the SAM-FS database and then retrieving the desired entry. Users are requested to submit a file recovery request only for critical, irreplaceable data.

In the event of a personal disaster, send email to help@nf.apac.edu.au including the following information:
  - user name
  - contact number
  - full pathname of lost file
  - date file was removed
  - best guess as to when file was created or last modified.

Projects dealing with massive amounts of data need to carefully consider all aspects of data acquisition, storage, retrieval, navigation, and interpretation. APAC National Facility consultants are available to assist projects in all phases of this cradle-to-grave process. Send email to help@nf.apac.edu.au to make an appointment for an exploratory conversation. Some topics on which projects might need assistance include:

1. project data flow;
2. determining if a formal database manager is recommended;
3. database organization;
4. efficient access of project data;
5. data interpretation tools;
6. source configuration management.

If your project changes direction or data patterns, let APAC National Facility staff know so we can adjust your storage environment appropriately.