User Study

Participant Demographics

To ensure the reliability and fairness of our user study, we carefully designed a participant pairing strategy based on self-reported metrics of programming background. Specifically, each experimental group (EG) participant (e.g., P1–P10) was paired with a control group (CG) counterpart (P11–P20) who exhibited comparable programming skill, Python familiarity, and education level.

This matched-pairing design minimizes individual-level variance and allows us to isolate the effect of explanation in semantic code search performance.

User Study Interface

Interface of Experimental Group user study

In this interface, participants are shown additional explainability information. For each recommended code snippet, the interface highlights which concepts in the query align with specific parts of the code, helping users understand why the code was retrieved.

Interface of Control Group user study

This interface simulates a traditional code search experience. It presents the same queries and retrieved code snippetsas the experimental group but without any explanation or concept-level alignment information.

Task Types & Representative Examples

We categorize the 20 study queries into five coarse-grained daily programming task types. For each type, we show one representative example using two screenshots: the query and its ground-truth code.

Data I/O

Read/write data from files/directories/databases; persist results into CSV/JSON/DB.

Query

Code

Data Conversion

Convert values to target types/formats to meet downstream interface expectations.

Query

Code

Parsing & Extraction

Parse or extract information from streams, attributes, timestamps, or binary headers.

Query

Code

Data Validation

Validate inputs/data against constraints, rules, and allowed sets.

Query

Code

System and Configuration Management

Manage runtime states/configurations, render templates, refresh routes, and present outputs.

Query

Code

Threats to Validity

Sample Size and Diversity
While we recruited 20 participants from reputable institutions and ensured diversity in education levels and programming backgrounds, the relatively small sample size may limit the generalizability of the findings. Future studies with larger and more varied populations would strengthen external validity.
Self-Reported Proficiency Matching
Participant pairing was based on self-assessed programming skill and Python familiarity. Such subjective assessments may not perfectly reflect actual competence, potentially introducing pairing imbalances that affect internal validity.
Lack of Long-Term Evaluation

The study measured short-term task performance under time-limited, artificial conditions. It remains unclear whether the advantages of the explanation interface would persist in longer-term, real-world development scenarios.

Page updated

Google Sites

Report abuse