7
$\begingroup$

For my thesis, I have created a development approach (consisting of schemes, an application template and prototype code) that should make it easier to develop and adapt applications for a specific platform.

I have conducted interviews with developers of that platform to evaluate my approach. There have been 5 interviewees so far and the feedback that I got from them is sufficient for my scope, I would say.

However, in my thesis I would like to justify my decision that 5 evaluators are enough. Is there any scientific research or paper that suggests a certain number of evaluators for such a rather theoretical approach?

I know Nielsen's work about heuristic usability evaluation which says that with 5 participants, you discover around 75% of all problems in software usability / UI testing.

But as I said, my approach is no specific software product that has to be evaluated, but only the model itself, which is why I'm looking for research in that area.

Improve this question
$\endgroup$
3
  • $\begingroup$ it would seem to be related to the complexity of the software and the "coverage" of users choosing the functions of the software (similar to "test coverage" except users, not code, seen as testers). you could have a single user choose all functions and get high "coverage", or have many users that do not "cover" all the functions that much. so you might want to switch to a different angle focusing on all the functions of your code & which ones were actually "touched" (tested). $\endgroup$ – vzn Sep 1 '13 at 15:53
  • $\begingroup$ Are your interviews of qualitative or quantitative nature? $\endgroup$ – Raphael♦ Sep 2 '13 at 10:25
  • $\begingroup$ They were more qualitative. $\endgroup$ – saschoar Sep 21 '13 at 1:29
5
$\begingroup$

"Is there any scientific research or paper that suggests a certain number of evaluators for such a rather theoretical approach?"

There is a statistical method to calculate the correct sample size for an experiment.
This method is called a "power analysis" [1]. It calculates the minimum sample size needed to verify that an experiment's results are statistically significant. This approach assumes that you have at least two groups. A control group and an experimental group (i.e. control group uses the platform without your modifications, experimental group uses the platform with your modifications).

Thus, while there is a scientific process to calculate the correct sample size this approach is probably overkill for your study. Nielsen [2] states that heuristic evaluation is just a "cheap and quick" method for easy evaluation. As such, I believe you can justify a sample size of 5 by citing Nielsen. However, if you do want to standardize your evaluation process and use a statistical approach I encourage you to look into power analysis.

You may be interested in this online tool for calculating power.

  1. http://en.wikipedia.org/wiki/Statistical_power
  2. http://www.nngroup.com/topic/heuristic-evaluation/
Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.