A permutation test based on regression error characteristic curves for software cost estimation models

作者：Mittas Nikolaos; Angelis Lefteris^*

来源：Empirical Software Engineering, 2012, 17(1-2): 34-61.

DOI：10.1007/s10664-011-9177-5

• Cost prediction models
• Accuracy measures
• Graphical comparison
• Hypothesis test
• Permutation test
• REC curves

Background Regression Error Characteristic (REC) curves provide a visualization tool, able to characterize graphically the prediction power of alternative predictive models. Due to the benefits of using such a visualization description of the whole distribution of error, REC analysis was recently introduced in software cost estimation to aid the decision of choosing the most appropriate cost estimation model during the management of a forthcoming project. %26lt;br%26gt;Aims Although significant information can be retrieved from a readable graph, REC curves are not able to assess whether the divergences between the alternative error functions can constitute evidence for a statistically significant difference. %26lt;br%26gt;Method In this paper, we propose a graphical procedure that utilizes (a) the process of repetitive permutations and (b) and the maximum vertical deviation between two comparative Regression Error Characteristic curves in order to conduct a hypothesis test for assessing the statistical significance of error functions. %26lt;br%26gt;Results In our case studies, the data used come from software projects and the models compared are cost prediction models. The results clearly showed that the proposed statistical test is necessary in order to assess the significance of the superiority of a prediction model, since it provides an objective criterion for the distances between the REC curves. Moreover, the procedure can be easily applied to any dataset where the objective is the prediction of a response variable of interest and the comparison of alternative prediction techniques in order to select the best strategy. %26lt;br%26gt;Conclusions The proposed hypothesis test, accompanying an informative graphical tool, is more easily interpretable than the conventional parametric and non-parametric statistical procedures. Moreover, it is free from normality assumptions of the error distributions when the samples are small-sized and highly skewed. Finally, the proposed graphical test can be applied to the comparisons of any alternative prediction methods and models and also to any other validation procedure.

• 出版日期2012-2