1- Contacting the experienced developers. We shortlisted the experienced programmers based on their interests and skills. We used GitHub and LinkedIn to contact them. The participants included the original developers of an industry project, contributors of open source systems contacted via the information extracted from the commits (and our previous collaborations), experienced programmers based on our prior collaborations (including I-Corps), and interested developers who are using third party code of these open source systems hired via LinkedIn. We fount 32 developers who were interested to participate in our study.

3- Based on the results of the pre-study survey, we assigned each participant a software system and a refactoring tool in a counterbalanced manner. Some participants were assigned two system/tool pairs. For these subjects, we made sure to assign both interactive and automated refactoring tools.

5- We asked the participants to evaluate the assigned tool on the selected systems based on three different executions. As specified in the video demo of our IMMO tool on the home page, the data of the interactions with the tool (accept or reject of the refactorings) is automatically saved in a log file. The same feature exists in all the competitor interactive tools (Alizadeh et al. and Mkaouer et al.). Ouni et al. and JDeodorant tools have a feature to export the results in a csv format then commenting the Refactorings one by one to give their opinion to accept or reject.