Similar to this question, but I wanted to be more specific to my real-world situation.

I am a college software engineer currently working on a project that involves a large number of read cycles. Essentially, I have a bunch of text files of various sizes (i.e. ranging from 10 KB to 1 GB) that get read each time the code runs, and this code is run many, many times in a day for testing purposes as it is tweaked and improved - probably a couple hundred times per day.

Recently I began to wonder if data corruption might become a problem. I've read some literature that suggests a very small chance of read errors/bit flipping each time you read a file, so obviously if you read a lot of times, the practical chance of such a thing occurring gets higher. The files I am reading are pretty important and it would be an issue if they got corrupted or damaged in any way, and so far I've always been careful to back them up each day and take typical precautions in case any corruption occurs. However, I wonder if I could do more, and thus here is my question:

How realistic is a concern about data corruption when reading large numbers of files a large number of times, and in this case, is it even worth worrying about at all? Should I worry about it and/or take more steps to protect any important data files in the future, or is it just not a realistic concern? Would appreciate any help or advice on this matter, as I'm obviously not a very experienced developer yet - hopefully this isn't a silly question!

  • $\begingroup$ I don't think this is really a computer science question: it's about the practical operation of actual computer systems. I'm not sure where would be a good place to deal with this. However, I suggest you look at what Google does to cope with things like hardware failures: they're in an environment with whatever ridiculous number of machines, so components fail all the time and they have to be able to deal with that. $\endgroup$ Aug 7 '19 at 19:57
  • $\begingroup$ @DavidRicherby That is a good suggestion, thank you. Sorry if this is kind of a weird question, I'm more of a software guy so I don't really know if this is even something people worry about or not when it comes to computer hardware. Figured I would just ask and see if anyone had some advice. $\endgroup$
    – Sciborg
    Aug 7 '19 at 20:16
  • $\begingroup$ It's a very good question -- not weird at all. I'm just not sure it's on-topic. $\endgroup$ Aug 7 '19 at 20:34

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.