Evaluate Quality of Solutions in Pareto-based SBSE
M. Li, T. Chen, X. Yao. How to evaluate solutions in Pareto-based search-based software engineering? A critical review and methodological guidance. IEEE Transactions on Software Engineering, 48(5), 2022.
M. Li, T. Chen, X. Yao. How to evaluate solutions in Pareto-based search-based software engineering? A critical review and methodological guidance. IEEE Transactions on Software Engineering, 48(5), 2022.
M. Li, T. Chen, X. Yao. A critical review of "A practical guide to select quality indicators for assessing Pareto-based search algorithms in search-based software engineering": Essay on quality indicator selection for SBSE. The 40th International Conference on Software Engineering (ICSE): New Ideas and Emerging Results Track, 17-20, 2018.
M. Li, T. Chen, X. Yao. A critical review of "A practical guide to select quality indicators for assessing Pareto-based search algorithms in search-based software engineering": Essay on quality indicator selection for SBSE. The 40th International Conference on Software Engineering (ICSE): New Ideas and Emerging Results Track, 17-20, 2018.
Brief:
Brief:
Multi-objective optimisation stays relatively new to SE researchers, let alone evaluating the outcome of optimisation algorithms. Simply following the evaluation methods for general multiobjective optimisation problems may not be appropriate for specific SE problems, especially when the problem nature or decision maker’s preferences are explicitly/implicitly available. This has been well echoed in the literature by various inappropriate/inadequate selection and inaccurate/misleading use of evaluation methods. This paper carries out a systematic and critical review of quality evaluation for multiobjective optimisation in SBSE, and then provides a methodological guidance for selecting and using evaluation methods in different SBSE scenarios.
Multi-objective optimisation stays relatively new to SE researchers, let alone evaluating the outcome of optimisation algorithms. Simply following the evaluation methods for general multiobjective optimisation problems may not be appropriate for specific SE problems, especially when the problem nature or decision maker’s preferences are explicitly/implicitly available. This has been well echoed in the literature by various inappropriate/inadequate selection and inaccurate/misleading use of evaluation methods. This paper carries out a systematic and critical review of quality evaluation for multiobjective optimisation in SBSE, and then provides a methodological guidance for selecting and using evaluation methods in different SBSE scenarios.
Observations:
Observations:
Directly using quality indicators in multi-objective optimisation to compare solution sets in specific SE scenarios may result in misleading conclusions.
Directly using quality indicators in multi-objective optimisation to compare solution sets in specific SE scenarios may result in misleading conclusions.
An example that no consideration of contextual information may give unwanted evaluation results. Considering two solutions sets A and B for optimising the code coverage and the cost of testing time on the software test case generation problem, B is evaluated better than A on eight frequently-used indicators: GD(B) = 0.02 < GD(A) = 0.26, ED(B) = 0.5 < ED(A) = 0.89, epsilon(B) = 0.1 < epsilon(A) = 0.3, GS(B) = 0.15 < GS(A) = 0.46, PFS(B) = 5 > PFS(A) = 4, IGD(B) = 0.02 < IGD(A) = 0.27, HV (B) = 0.77 > HV (A) = 0.43, C(B) = 0.8 > C(A) = 0.25. However, the decision maker may be more interested in A (specifically solution (450, 1.0)) if they favor the full code coverage and then possible low cost.
Guidance:
Guidance:
General procedure of quality evaluation in Pareto-based SBSE
General procedure of quality evaluation in Pareto-based SBSE
