# How are statistics being applied in computer science to evaluate accuracy in research claims?

I have noticed in my short academic life that many published papers in our area sometimes do not have much rigor regarding statistics. This is not just an assumption; I have heard professors say the same.

For example, in CS disciplines I see papers being published claiming that methodology X has been observed to be effective and this is proved by ANOVA and ANCOVA, however I see no references for other researchers evaluating that the necessary constraints have been observed. It somewhat feels like as soon as some 'complex function and name' appears, then that shows the researcher is using some highly credible method and approach that 'he must know what is he doing and it is fine if he does not describe the constraints', say, for that given distribution or approach, so that the community can evaluate it.

Sometimes, there are excuses for justifying the hypothesis with such a small sample size.

My question here is thusly posed as a student of CS disciplines as an aspirant to learn more about statistics: How do computer scientists approaches statistics?

This question might seems like I am asking what I have already explained, but that is my opinion. I might be wrong, or I might be focusing on a group of practitioners whereas other groups of CS researchers might be doing something else that follows better practices with respect to statistics rigor.

So specifically, what I want is "Our area is or is not into statistics because of the given facts (papers example, books, or another discussion article about this are fine)". @Patrick answer is closer to this.

• Please let me know if my answer isn't quite what you were looking for... I can find some examples, if you would like, but it sounds like you already have plenty yourself. Plus, unless you have access to proceedings, it might be tough to present links to the papers. – Patrick87 Apr 7 '12 at 2:38
• I have access to IEEE and ACM digital libraries, examples would be very well welcomed as well on your post, I could use them, later on when I earn my PhD to show the students I end up advising good and bad examples regarding this point. And yes, this is the sport of opinion and statement I was looking for. – Oeufcoque Penteano Apr 7 '12 at 2:41
• Note that huge parts of computer science are not empirical at all and therefore to not (ab)use statistics. On a snarky note, software engineering people are "known" to write much and say little (of relevance). Step one would be to learn statistics; few of the aspiring software engineering researchers I know have ever taken a course or anything on statistics. – Raphael Apr 7 '12 at 12:37
• Good point. I am used to listen to this as well. What would it be considered next? – Oeufcoque Penteano Apr 7 '12 at 16:06

As a graduate student in computer science, who has exposure to research in fields other than computer science, and whose research group works in an area of computer science where statistics can be fruitfully applied, I can offer my experience; your mileage may vary.

In general, even the most well-meaning scientific research can fail to rigorously apply statistical analysis to results, and it is my experience that this does not always preclude papers including such poorly-analyzed results from being accepted for publication. The area in which my group operates is mainly in distributed computing and high-performance computer architecture. Often, research involves experimental designs the performance of which cannot easily be understood analytically in the required detail. As such, empirical results are often used as evidence for claims.

Clearly, experiments should be design - and results analyzed - in such a way as to provide some confidence that the results are statistically significant. Most of the time, this is not done, even in some of the most important venues. When statistical analysis is applied, it is almost never rigorous in any meaningful sense; the most one typically sees (and one is glad to see it!) is that an experiment was repeated n times, for some arbitrarily-chosen n, where typically $1 < n < 5$. The selection of error bars (if any are indicated) seems to be mainly a matter of personal preference or taste.

In summary, no, it's not just you; and it's not just software engineering. In general, based on my experience, several areas of computing research seem to err on the side of not doing enough. Indeed, it might be even be detrimental to the viability of a submitted paper to dwell on statistical considerations. That's not to say that I find the situation satisfactory; far from it. But these are my impressions. For instance, you may take a look at section 5 of this paper, which was presented at Supercomputing 2011, one of the most high-profile conferences in the area of high-performance computing. Specifically, take a look at some of the discussion of results in section 5, and see whether you arrive at the same conclusions that I do about the rigor of statistical analysis of experimental results.

More generally, this shortcoming may be symptomatic of a condition in some areas of computing to publish more papers rather than fewer, to target conferences rather than journals, and to emphasize incremental progress rather than significant and fundamental improvements in understanding. You might consult this article, which provides valuable insight along these lines.

• Thank you very much for your opinion. I'm glad to see that other people think in this way. I hope the community have other opinions to share as well or on the very least vote to state opinion. :) Too bad here I'm still a rookie so I can't up vote anything. – Oeufcoque Penteano Apr 7 '12 at 2:39
• @OeufcoquePenteano Hey, everybody's a rookie when they start out. This is a good question and raises an interesting point about a potential shortcoming in computing research standards. I will add to the answer shortly. Also note that this question may be closed (sooner rather than later, even) by other members, since questions like this have (so far) been regarded with some suspicion, and since this site is still in beta, the community is still trying to define what is on- and off-topic here. – Patrick87 Apr 7 '12 at 2:51
• But the initial proposal mentioned about computer science and research and this one meet this criteria right? At this point there is no other more refined criteria that would kick this question out of the game, or there would be? Or is there any other stack family that I would be able to discuss on this question? Thank you for your compliment and attention on replying it :) – Oeufcoque Penteano Apr 7 '12 at 3:16
• As for benchmarking proof-of-concept implementations, I struggle to see how significance can be established. It does not seem possible to apply statistics here; taking measurements alone is not statistics. It is important that the measurements are interpreted carefully and with perspective, though. – Raphael Apr 7 '12 at 12:51
• @Patrick87: I do not want to advocate ignoring statistical significance. It is hard, though. Incidentally, I do benchmarks for my master thesis. I run my stuff (many times) on two to three machines and take care to measure relevant times. What else can I do? I can't even begin to employ rigorous statistics because I have no idea what a stochastic model for multi-core execution of algorithms with the JVM in the middle may be. – Raphael Apr 7 '12 at 13:30

Software engineering includes many features. Two of them are human factor and quality measure.

Let’s say I want to do productivity analysis. The data collection would be hard comparing to algorithm analysis because the data is about human productivity. Also the objective measure of quality is not easy to achieve.

10 lines of code per day for an avionics system versus 150 lines of code per day for an app on smart phones, which one has higher productivity and which one has better quality? And if both claim that they are using the same methodology? Comparing them is to compare apples and oranges.

Sometimes it is hard to achieve accurate measure of code efficiency. For example, I put in a bunch of unuseful variables and lots of lines of code for those variables, say for debugging purpose. This boosts my productivity at development stage. At the end, I take out all of them and I say I improve my code to achieve efficiency.

Later on, a researcher comes in and perform efficiency analysis. He might treat the above as noises and only concentrate on the final results. Some researchers pay attention to the noises. Then you are going to see articles with different conclusions.

Statistics is supposed to be a tool to assist the researchers in finding causes of issues. Many researchers use it to draw conclusions. This is what you have observed.

Some remarks above might lead the OP to think that I am against the usage of stats in software engineering. If so, I'd like to make myself clear.

I am not against stats. Stats analysis can tell you X might be true. But, that should not be the end of the research. The next task should be to find out if X is actually true and why. This is what I believe science is about - to find the truth.

Whether or not software engineering belongs to computer science is another issue.

• Hello, thanks. I think you are able to comment on your on question so this should work. You mean that software engineers can just be allowed to be more informal on statistics because of the uncertainties? I don't know. See, I have a book I got from a professor that teaches statistics for social researchers. I think their side is way worse than ours, and they even have well defined examples in a book for that. Why,we, even have static code to try different approaches to it would not be able to? – Oeufcoque Penteano Apr 7 '12 at 5:23
• I wonder if this is the gap I usually hear about between research and people who actually work in industry. I agree with you that for the developer side you are not concerned about statistics, but getting the program working. But, yes, the project manager would need it to predict if the developer would be behind the schedule or not, maybe? Still considering the 'research claims' part of the question, do you believe accuracy in statistics would be useless as for example inferring that few things developers do unconsciously reduce their productivity and enjoyment in work? My concern lies there. – Oeufcoque Penteano Apr 7 '12 at 5:49
• You mean that going on experimental research would consider people as subjects? I was talking about measuring code to understand this, not experimenting with people. Sorry, I am confused only your last statement. But, thanks for your insights! – Oeufcoque Penteano Apr 7 '12 at 6:06
• Alright, I understand now. Since I'm a rookie on this I didn't know that long discussions on comments would be a bad practice so I will stop at this point. Thank you for your insights. I will wait a few more days and elect an answer for it (in case the question is not appropriated for this community and gets closed). – Oeufcoque Penteano Apr 7 '12 at 6:25
• I wonder wether you are talking about the same statistcics. In general public, a "statistic" is usually a bunch of measurements. In maths and science, "statistics" the estimation of model parameters given data/measurements. Therefore, listing LOC counts is not even statistics in the scientific sense; if you can (accurately) estimate the parameter "average developer coffee consumption" from LOC data, then you are doing (relevant) statistics; you have found correlating events. – Raphael Apr 7 '12 at 12:49

Statistics is hard, and often counter-intuitive. Besides, the urge to "do one more experiment" to see if there is an effect (and stop when it shows up) is strong, specially if the experiments are costly (time, work, not just money). Also remember that publishing a paper on how the carefully set up, long and costly experiment shows no statistically significant relationship tends to be impossible.

Specially in software engineering there are many uncontrollable variables. To account for them you'd need many replications of the experiment, and you get resources to do one or at best two.

My question here is thusly posed as a student of CS disciplines as an aspirant to learn more about statistics: How do computer scientists approaches statistics?

there are several questions above & some not the same as the title question & in some ways this question has an underlying faulty premise/misconception about some lack of connection between statistics and CS. the general question is about the interface of computer science and statistics.

there is a vast, verging on intense overlap in some areas and it is an increasing trend with the new strongly emerging field of big data. at some schools (eg even elite "Ivy League" schools) the CS degree is tightly coupled with the mathematics and statistics departments and some have a joint major. there is a very strong interconnection in the CS/statistical field of Machine Learning. also the relatively new field of bioinformatics has very strong CS+statistical grounding.

there is an entire field Computational statistics focused on the interface!

Computational statistics, or statistical computing, is the interface between statistics and computer science. It is the area of computational science (or scientific computing) specific to the mathematical science of statistics. This area is also developing rapidly, leading to calls that a broader concept of computing should be taught as part of general statistical education.[1]

yes, agreed, as pointed out in the question, there are many CS papers that dont use statistics, including situations (such as evaluating empirical experiments) where it might be even highly applicable & relevant, but exactly the same can be said of many other scientific fields, eg mathematics, and even more applied fields like physics.

there are many ways to use/apply statistics, some less rigorous than others, and not all contexts call for full application of the very advanced aspects of statistics. for example just running multiple experiments and plotting error bars for statistical deviation (or even merely averages!) is a basic use of statistics. more rigorous uses include hypothesis testing, but there is a general observation in the field that many scientific papers do not do rigorous hypothesis testing even where it might be applicable.

also, this question is tagged with . this was my major, and a statistics class was required to pass to complete this degree at my school & to get an engineering-certified major (eg ABET), this is likely the case at many other universities. if one wants more applied and rigorous CS type principles such as applications of statistics, one can go the "software-engineering" route in education.

• as pointed out in another comment, uses of statistics even in more abstract TCS include average case complexity analysis and PAC learning etc – vzn Feb 8 '14 at 17:44
• Average-case complexity analysis does not use statistics. Stochastics, certainly, but that's another thing. – Raphael Feb 9 '14 at 22:55
• strongly disagree! it plainly uses key/fundamental statistical concepts but perhaps is not taught that it does! academic views of the fields tend to be compartmentalized & its reflected here! imho any ref to a distribution of inputs etc is obviously an invocation of statistical concepts... – vzn Feb 9 '14 at 23:05