Evaluating different i-based approaches for selecting functional requirements while balancing and optimizing non-functional requirements: A controlled experiment - Zubcoff, Jose; Garrigós, Irene; Casteleyn, Sven; Mazón, Jose-Norberto; Aguilar, Jose-Alfonso; Gomariz-Castillo, Francisco (Information and software technology, 2019)
•When modelling, subjects may perform better using alternative i*-derived models and visualizations for particular tasks.•As models become complex, Pareto front based visualization outperforms original i* notation when selecting functional requirement to implement while balancing non-functional requirements.•It is 3.51 times more likely to solve modelling tasks correctly using Pareto front tabular or radar chart visualization.•Subjects solve complex modelling tasks quicker using Pareto front tabular (around 10% faster) or radar chart visualization (around 30% faster).•Pareto front tabular produces more correct answer; radar chart visualization is faster.•Experience helps in solving the exercises quicker, but has no effect on amount of correct answers.•Visual or textual learning preference has no effect on score. A relevant question in requirements engineering is which set of functional requirements (FR) to prioritize and implement, while keeping non-functional requirements (NFR) balanced and optimized. We aim to provide empirical evidence that requirement engineers may perform better at the task of selecting FRs while optimizing and balancing NFRs using an alternative (automated) i* post-processed model, compared to the original i* model. We performed a controlled experiment, designed to compare the original i* graphical notation, with our post-processed i* visualizations based on Pareto efficiency (a tabular and a radar chart visualization). Our experiment consisted of solving different exercises of various complexity for selecting FRs while balancing NFR. We considered the efficiency (time spent to correctly answer exercises), and the effectiveness (regarding time: time spent to solve exercises, independent of correctness; and regarding correctness of the answer, independent of time). The efficiency analysis shows it is 3.51 times more likely to solve exercises correctly with our tabular and radar chart visualizations than with i*. Actually, i* was the most time-consuming (effectiveness regarding time), had a lower number of correct answers (effectiveness regarding correctness), and was affected by complexity. Visual or textual preference of the subjects had no effect on the score. Beginners took more time to solve exercises than experts if i* is used (no distinction if our Pareto-based visualizations are used). For complex model instances, the Pareto front based tabular visualization results in more correct answers, compared to radar chart visualization. When we consider effectiveness regarding time, the i* graphical notation is the most time consuming visualization, independent of the complexity of the exercise. Finally, regarding efficiency, subjects consume less time when using radar chart visualization than tabular visualization, and even more so compared to the original i* graphical notation. [Display omitted]