# What empirical evidence do we have for or against a correlation between fault density and LOC?

LOC = lines of code

KLOC = Thousand lines of code

Fault (or defect) density = number of reported bugs per line of code.

Software artifact = function, class, module

Reading research papers on fault density and fault prediction, it seems a bit hard to get an overview, because there are lots of studies, and lots of different technologies used, both statistics and machine learning.

The rational behind a correlation between fault density and LOC is that with higher LOC of a software artifact comes higher complexity, and it gets harder for the programmer to change the code; changeability decreases, which leads to more bugs. Note that this is not a question about a correlation between fault density and complexity, even if there might be such a connection too.

I guess what I'm looking for is a study or a book that has an overview over current research in this topic. :)

Here's one study claiming there is such a correlation: https://www.gwern.net/docs/dual-n-back/1997-hatton.pdf

Edit: Google Scholar has an option to search for only review articles, so I'll dig into this. Will post anything interesting.

Edit 2: One review article: https://media.neliti.com/media/publications/90270-EN-a-systematic-literature-review-of-softwa.pdf

However, even though various defect prediction methods have been proposed, but none has been proven to be consistently accurate (Challagulla et al., 2005) (Lessmann et al., 2008). The accurate and reliable classification algorithm to build a better prediction model is an open issue in software defect prediction. There is a need for an accurate defect prediction framework which has to be more robust to noise and other problems associated with on datasets

• Here's another one: Some Misconceptions about Lines of Code (1997). Claims there is a negative correlation but this is a statistical artifact -- variables X and Y / X are always negatively correlated even if Y and X are independent -- and has nothing to do with longer files being less buggy.
– 6005
Oct 6 '21 at 18:12
• It's a great, interesting question! I don't know the answer, but software engineering conferences like FSE, ICSE, ASE, and ISSTA are probably good places to look for answers.
– 6005
Oct 6 '21 at 18:13
• @6005 Hm, thanks. Oct 6 '21 at 18:41
• @6005 The study I link in my last edit mentions this effect, actually. Oct 10 '21 at 9:38

Not an easy question to answer, because LOC can mean many different things:

• A large function
• A large class (or module in FP)
• A large library
• A large file
• A large code-base

In some of these cases (function and class), LOC can be a proxy for complexity [citation needed]. You can also normalize complexity based on size.

The Software fault prediction metrics: A systematic literature review paper contains a section on LOC in fault prediction research, which I will quote fully below. The conclusion is in the last sentence:

the overall effectiveness of the LOC metric was estimated as moderate

The comparison to other metrics is the interesting part, however.

(A description of the different metrics is available in appendix C.)

The first metrics are complexity measures that work on all languages. The middle part is metrics only applicable to object-oriented code. The last part is metrics that relate to process and change in the code.

It's interesting to note the effectiveness of metrics in the pre- and post-release scenarios. LOC is noted as being more predictive before the release.

Full section:

4.3.1. RQ2.1: Are size metrics useful for fault prediction?

The simplest, the easiest to extract and the most frequently used metric, i.e. LOC, is still being discussed to this day. There are many studies investigating the relationship between lines of code and number of faults. The simplest studies have ranked the modules according to their size to find out whether a small number of large modules are responsible for a large proportion of faults. E.g. in Zhang [151] three versions of Eclipse were used to investigate pre-release and post-release ranking ability of LOC at the package level. This study showed that 20% of the largest modules were responsible for 51–63% of the defects.

Ostrand et al. [130] used the negative binomial regression model on two large industrial systems. In the simple model, using only LOC, the percentage of faults, contained in the 20% of the files that were the largest in terms of the number of lines of code, was on average 73% and 74% for the two systems. In a richer model, where other metrics were used, the top-20% of files ordered by fault count contained, on average, 59% of the lines of code and 83% of the faults. The top-20% of files contained many large files, because the model predicted a large number of faults in large files. In analyzing which files were likely to contain the largest number of faults relative to their size, they used the model’s predicted number of faults and the size of each file to compute a predicted fault density. The top-20% of files contained, on average, only 25% of the lines of code and 62% of the faults. Sorting files by predicted fault density was not as effective as sorting files according to fault count at finding large numbers of faults, but it does result in considerably less code in the end.

Fenton and Ohlsson [84] investigated, among many hypotheses, the Pareto principle [28] and the relationship between size metrics and the number of faults. They used a graphical technique called the Alberg diagram [127] and two versions of a telecommunication software. As independent variables LOC, McCabe’s cyclomatic complexity and SigFF metrics were used. In pre-release 20% of the modules were responsible for nearly 60% of the faults and contained just 30% of the code. A replicated study by Andersson and Runeson [59] found an even larger proportion of faults, in a smaller proportion of the modules. This result is also in agreement with [130,151]

Fenton and Ohlsson also tested the hypothesis of whether size metrics (such as LOC) are good predictors of pre-release and post-release faults in a module and whether they are good predictors of a module’s pre-release and post-release fault density. They showed that size metrics (such as LOC) are moderate predictors of the number of pre-release faults in a module, although they do not predict the number of post-release failures in a module, nor can they predict a module’s fault density. Even though the hypothesis was rejected, the authors concluded that LOC is quite good at ranking the most fault-prone modules. Andersson and Runeson, on the other hand, got varying results. The first two projects did not indicate any particularly strong ranking ability for LOC. However, in the third project, 20% of the largest modules were responsible for 57% of all the faults

Koru et al. [35,104] reported that defect proneness increases with size, but at a slower rate. This makes smaller modules proportionally more problematic compared with larger ones

In [34,59], it is noted that relating size metrics to fault density may be misleading, as there will always be a negative correlation between size metrics and fault density, because of the functional relationship between the variables [47]. However, no studies using fault density as dependent variable were excluded from the review because we wanted to represent the entire research field. In this section, studies using fault density are compared with LOC size ranking ability to assess the LOC predictive capabilities

To some extent, size correlates with the number of faults [130,151,154,112,83,78], but there is no strong evidence that size metrics, like LOC, are a good indicator of faults [84,59]. A strong correlation was observed between the size metric LOC and fault proneness in pre-release and small studies, while in post-release and large studies, only a weak association was found (Table 5). This may indicate that studies with lower validity (smaller data sets) gave greater significance to the LOC metric than studies with high- er validity (larger data sets). Therefore, the reliability of the studies in terms of study quality assessment was taken into account and the overall effectiveness of the LOC metric was estimated as moderate.

So there you go. :)

• Please make sure to give any advice on actions that should be taken based on this. Without any actions, it's quite pointless, right? Oct 12 '21 at 15:23
• @gnasher729 I think the interesting action is whether or not you can expect bugs to decrease in your code-base if you limit the size of functions and classes. I think the answer is "maybe to some degree". Same question applies to limit complexity: "probably less than a size limit", based on the linked review article. But try to limit the churn rate might have even better results. :) Oct 12 '21 at 15:39

There are several factors that affect both number of faults and lines of code, but sometimes in opposite directions.

You can increase the number of lines of code by writing code that is simple, clear, contains self-tests, is well-documented, handles border cases correctly etc, and this will be more code with fewer bugs.

You can increase the number of lines of code by not understanding the problem and writing overly complicated code that just works by coincidence in many cases, and any bug found creates changes that make the code even bigger and worse. So you will have more code with more bugs.

And keep in mind that we can never correlate between code size and errors - we can only correlate between code size and detected errors! Which are also related in a strange way. More errors will produce more detected errors. But more detected errors will lead to errors getting fixed and therefore the number of errors going down.

• Please make sure to cite sources. Oct 10 '21 at 18:25
• Why? Anything here that cannot be understood with a bit of brain? Oct 12 '21 at 15:27