1- Contacting the experienced developers. We shortlisted the experienced programmers based on their interests and skills. We used GitHub and LinkedIn to contact them. The participants included the original developers of an industry project, contributors of open source systems contacted via the information extracted from the commits (and our previous collaborations), experienced programmers based on our prior collaborations (including I-Corps), and interested developers who are using third party code of these open source systems hired via LinkedIn. We fount 32 developers who were interested to participate in our study.
2- All developers were given a pre-study questionnaire to fill.
Show Pre-study Survey and Results
3- Based on the results of the pre-study survey, we assigned each participant a software system and a refactoring tool in a counterbalanced manner. Some participants were assigned two system/tool pairs. For these subjects, we made sure to assign both interactive and automated refactoring tools.
4- We had a video-conference with each developer to explain the tasks, show a demo of different tools, and clarifying the process of result submission. The participants attended a live streaming lecture using bluejeans and they all have high experience on refactoring and the used open source systems. We tried to mitigate possible threats by enabling both offline and online questions, recorded video of the lectures, doing two exercises to make sure that they are able to evaluate refactorings, etc.
5- We asked the participants to evaluate the assigned tool on the selected systems based on three different executions. As specified in the video demo of our DOIMR tool on the home page, the data of the interactions with the tool (accept or reject of the Refactorings) is automatically saved in a log file. The same feature exists in all the competitor interactive tools (Alizadeh et al. and Mkaouer et al.). Ouni et al. and JDeodorant tools have a feature to export the results in a CSV format, then commenting the Refactorings one by one to give their opinion to accept or reject.
6- For the developers who were not assigned to experiment our proposed tool, we asked to evaluate our tool on a software system of their own choice. This step is done to make sure that every participant has tried our tool to be able to fill the post study questionnaire.
7- Post-study questionnaire is completed by all developers.
Show Post-study Survey and Results