A controlled experiment on time pressure and confirmation bias in functional software testing

• Published in: Empirical software engineering : an international journal Vol. 24; no. 4; pp. 1727 - 1761
• Main Authors: Salman, Iflaah, Turhan, Burak, Vegas, Sira
• Format: Journal Article
• Language: English
• Published: New York Springer US 01.08.2019; Springer Nature B.V
• Subjects: Bias; Compilers; Computer Science; Design; Design of experiments; Human behavior; Interpreters; Pressure effects; Programming Languages; Psychology; Requirements specifications; Software Engineering/Programming and Operating Systems; Software quality; Software testing; Software testing; Cognitive biases; Experiment; Human factors; Time pressure; Test quality; Confirmation bias
• ISSN: 1382-3256; 1573-7616
• DOI: 10.1007/s10664-018-9668-8

Context Confirmation bias is a person’s tendency to look for evidence that strengthens his/her prior beliefs rather than refutes them. Manifestation of confirmation bias in software testing may have adverse effects on software quality. Psychology research suggests that time pressure could trigger confirmation bias. Objective In the software industry, this phenomenon may deteriorate software quality. In this study, we investigate whether testers manifest confirmation bias and how it is affected by time pressure in functional software testing. Method We performed a controlled experiment with 42 graduate students to assess manifestation of confirmation bias in terms of the conformity of their designed test cases to the provided requirements specification. We employed a one factor with two treatments between-subjects experimental design. Results We observed, overall, participants designed significantly more confirmatory test cases as compared to disconfirmatory ones, which is in line with previous research. However, we did not observe time pressure as an antecedent to an increased rate of confirmatory testing behaviour. Conclusion People tend to design confirmatory test cases regardless of time pressure. For practice, we find it necessary that testers develop self-awareness of confirmation bias and counter its potential adverse effects with a disconfirmatory attitude. We recommend further replications to investigate the effect of time pressure as a potential contributor to the manifestation of confirmation bias.