A Controlled Experiment for Program Comprehension through Trace Visualization

• Published in: IEEE transactions on software engineering Vol. 37; no. 3; pp. 341 - 355
• Main Authors: Cornelissen, B, Zaidman, A, van Deursen, A
• Format: Journal Article
• Language: English
• Published: New York IEEE 01.05.2011; IEEE Computer Society
• Subjects: Added value; Analysis; Case studies; Computer engineering; Computer programs; Computer Society; Control systems; controlled experiment; Design; Digital Object Identifier; Documentation; dynamic analysis; Eclipses; Experiments; Gain measurement; Performance evaluation; Program comprehension; Programming; Reading comprehension; Scalability; Software; Software development; Software engineering; Software maintenance; Studies; Tasks; Time measurement; Visualization
• ISSN: 0098-5589; 1939-3520; 2326-3881; 1939-3520
• DOI: 10.1109/TSE.2010.47

Software maintenance activities require a sufficient level of understanding of the software at hand that unfortunately is not always readily available. Execution trace visualization is a common approach in gaining this understanding, and among our own efforts in this context is Extravis, a tool for the visualization of large traces. While many such tools have been evaluated through case studies, there have been no quantitative evaluations to the present day. This paper reports on the first controlled experiment to quantitatively measure the added value of trace visualization for program comprehension. We designed eight typical tasks aimed at gaining an understanding of a representative subject system, and measured how a control group (using the Eclipse IDE) and an experimental group (using both Eclipse and Extravis) performed these tasks in terms of time spent and solution correctness. The results are statistically significant in both regards, showing a 22 percent decrease in time requirements and a 43 percent increase in correctness for the group using trace visualization.