What are efficient and accurate techniques for monitoring the recoverability and integrity of files in very large preservation archives?

In very large archives, the time taken to recompute checksums periodically (scrubbing) is substantial, perhaps taking more than all the available time depending on the read bandwidth available! Also, each access to a preserved file increases the risk of damage due to hardware or software failure. Tapes are most stable in a cold, dark place far from exposure to the hazards of data centers. Disks are most at risk when the read/write head is flying close to the medium. All approaches are probabilistic, so which are most efficient and accurate?

To give the problem specificity, let's assume a fixed probability of local single-bit errors for each medium (one probability for tape, another for disk, SSD, etc) during a standard time period, and ignore all other types of errors (loss of an entire volume, for instance). We can also assume a fixed read bandwidth for each medium.

  • 2
    $\begingroup$ Are you associated with such an archive? If so, can you amend your post with the technical details? $\endgroup$
    – Emre
    Commented Apr 24, 2012 at 22:49
  • 1
    $\begingroup$ Do you mean something beyond checksums? $\endgroup$ Commented Apr 25, 2012 at 7:27
  • 4
    $\begingroup$ This problem seems hard to answer in its present form. What (probabilistic) model(s) are we to assume for degradation/corruption of data? Without more details on that, no answer is going to be meaningful. If the question is really about what models can be used to inform monitoring techniques... it might be good to point that out in the question. $\endgroup$
    – Patrick87
    Commented Apr 25, 2012 at 14:51
  • 2
    $\begingroup$ As is, your question is way too broad. I think you'll need to narrow it down to something we can answer: provide specifics on what your archives look like and what kind of degradation you expect. Also, I think it's a good idea if you incorporate your comment into your question. Voted to close accordingly (will vote to reopen if you improve the question in the case it gets closed). $\endgroup$ Commented Apr 25, 2012 at 17:25
  • 2
    $\begingroup$ Sounds like you need some kind of error correcting code. $\endgroup$ Commented Apr 25, 2012 at 20:26

1 Answer 1


This is not an answer to the question per se, but an extended comment.

There was an interesting set of slides regarding CERN's Advanced Storage (CASTOR) at CHEP'2010. This is a large mixed media archive with some 25 PB of tape archives (ca. 2010). One of the presentations: "Tape archive challenges" highlights some of the issues that seem to question the assumptions set in the OP.

Tapes are most stable in a cold, dark place far from exposure to the hazards of data centers.

Not quite. Even if you have a controlled environment in the tape storage you should expect a very small percentage of the tapes to fail. Therefore it makes sense to keep repacking the data constantly. Also older tape drives may fail making the data on old tapes inaccessible.

The latter reason makes archivists keep copying tapes at a much higher rate than their physical life span would suggest. Marty Perlmutter, "The Lost Picture Show: Hollywood Archivists Can't Outpace Obsolescence".

Disks are most at risk when the read/write head is flying close to the medium.

When the disk is spinning it constantly monitors itself for errors due to wear and errors due to defects in the manufacturing process. While rotation increases the probability of an error it also through predictive failure analysis reduces the chance of an error to occur undetected. On the contrary, as a mechanical unit a disk that was stored on a shelf might just fail to spin up.

A probabilistic approach to consistency checking has to be aligned with the statistical thresholds set for different actions. Suppose that a probabilistic model says that with probability 50% there is one error for a 1000 cartridges in a tape library. Does it give any hint on the possible action but to check consistency for the complete archive in order to find the possibly damaged part so that it can be rebuilt.

Many papers with statistical models of reliability of computer systems based on queuing theory have this problem. While they provide formulas to quickly estimate parameters of computer systems they often rely on some generic assumptions that may or may not apply in a particular setting.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.