Interview Questions for Semi-structured Interviews

1. Can you describe your code auditing experience?

2. Could you walk us through a past code auditing project? (Please share your screen for a detailed example.)

3. What is your typical process for conducting code audits? Please illustrate with examples from the shared project. Do you follow a specific methodology or have a unique approach? (We’ll follow up with specific questions as needed.)

4. What key aspects do you focus on to understand the code effectively? (We’ll follow up with specific questions as needed.)

5. How do you identify vulnerabilities during code auditing? (We’ll follow up with specific questions as needed.)

6. How do you use ChatGPT in your code auditing process? Have you encountered any difficulties or challenges? (We’ll follow up with specific questions as needed.)

Results

We provide raw data for review purposes: https://anonymous.4open.science/r/codemap_data-865A

All personal information has now been removed. The dataset includes anonymous interview transcripts, qualitative analysis codebook, quiz questions and answers and grading rubic, and participants' anonymous responses.

How are the formative study results reported in our paper (“understanding process,” “challenges,” and “expectations") linked to the original interview data? The qualitative analysis report (manually analyzed by the team using professional qualitative analysis software, Atlas.ti, which allows users to assign qualitative codes to interview data manually and discuss them within the team) is in the anonymous repo, showing the link between participants’ quotes and our qualitative codes. These codes were used to derive our findings. Note that our interviews cover more nuanced contents than what is reported in the paper, due to page limits.

Qualitative analysis for user feedback on three tools: