Qualitative Analysis on RQ1 Screen Inconsistency Experiment
Qualitative Analysis on RQ1 Screen Inconsistency Experiment
Embedded Files
Why GUIPilot succeeds and GVT fails?
Why GUIPilot succeeds and GVT fails?
The left-hand side (LHS) figure presents the matching results from GVT, while the right-hand side (RHS) figure showcases those from GUIPilot.
In each figure, the original screen is displayed on the left, and the mutated screen appears on the right.
Red Boxes: Highlight widgets that could not be matched => will be reported as missing or extra.
Green Boxes: Denote widgets that are unaffected by the mutation.
Yellow Boxes: Widgets whose relative positions have changed due to the mutation but can still identify a match.
Black lines: Highlight certain matches reported by the approach.
# Widget Insertion
# Widget Insertion
GVT will mismatch the rest of the widgets if their positions are shifted.
GUIPilot correctly matches the shifted widgets
GVT will mismatch the rest of the widgets if their positions are shifted.
GUIPilot correctly matches the shifted widgets
GVT will mismatch the rest of the widgets if their positions are shifted.
GUIPilot correctly matches the shifted widgets
GVT will mismatch the rest of the widgets if their positions are shifted.
GUIPilot correctly matches the shifted widgets
# Widget Deletion
# Widget Deletion
GVT will mismatch the rest of the widgets if their positions are shifted.
GUIPilot correctly matches the shifted widgets
GVT will mismatch the rest of the widgets if their positions are shifted.
GUIPilot correctly matches the shifted widgets
GVT will mismatch the rest of the widgets if their positions are shifted.
GUIPilot correctly matches the shifted widgets
GVT will mismatch the rest of the widgets if their positions are shifted.
GUIPilot correctly matches the shifted widgets
# Widget Swap
# Widget Swap
GVT fails to match the widget with its swapping pair, therefore GVT will not report the swapping pattern but report individual missing or extra widget.
GUIPilot can still match the widget with its swapping pair.
GVT fails to match the widget with its swapping pair, therefore GVT will not report the swapping pattern but report individual missing or extra widget.
GUIPilot can still match the widget with its swapping pair.
GVT fails to match the widget with its swapping pair, therefore GVT will not report the swapping pattern but report individual missing or extra widget.
GUIPilot can still match the widget with its swapping pair.
GVT fails to match the widget with its swapping pair, therefore GVT will not report the swapping pattern but report individual missing or extra widget.
GUIPilot can still match the widget with its swapping pair.
When does GUIPilot Report False Positives?
When does GUIPilot Report False Positives?
We only show the matching results for GUIPilot.
The main reason for FP is: If the widget layout changes too significantly, GUIPilot will produce mismatches.
A remedy is to design a more robust similarity metric that does not rely on the relative position.
# Widget Insertion
# Widget Insertion
Example 1:
Example 2:
Example 3:
Example 4:
# Widget Deletion
# Widget Deletion
Example 1:
Example 2:
Example 3:
Example 4:
# Widget Swap. The reason is due to the implementation issues, we occasionally create overlaying widgets.
# Widget Swap. The reason is due to the implementation issues, we occasionally create overlaying widgets.
Example 1: We swap two pairs of widgets; however, they occasionally overlap. This overlapping causes one widget to incorrectly match with a widget from the other pair.
Example 2: When we swap two widgets, one widget overlays the other, occupying a large area and affecting the color of the underlying widget.
When does GUIPilot Report False Negatives?
When does GUIPilot Report False Negatives?
We only show the matching results for GUIPilot.
The main reasons for FN are:
(1) If the widget layout changes too significantly, GUIPilot will produce mismatches, the true missing/extra elements are wrongly paired with another irrelevant widget.
(2) If the inserted widgets are undetected by the object detector, they will not participate in the follow-up matching step and, therefore will be unreported.
A remedy is to design a more robust similarity metric that does not rely on the relative position.
# Widget Insertion
# Widget Insertion
Example 1: Mismatch
Example 2: Mismatch
Example 3: The object detector didn't identify the new inserted widget.
Example 4: The object detector didnt identify the new inserted widget.
# Widget Deletion
# Widget Deletion
Example 1: Mismatch
Example 2: Mismatch
Example 3: Mismatch
Example 4: Mismatch
# Widget Swap. The reason is due to swapping 2 same-looking widgets.
# Widget Swap. The reason is due to swapping 2 same-looking widgets.
# Text change. The reason is because OCR didnt recognize the single character in a widget.
# Text change. The reason is because OCR didnt recognize the single character in a widget.
# Color change. The reason is because our color mutation is too subtle.
# Color change. The reason is because our color mutation is too subtle.
Why can't we outsource the inconsistency checking to VLM?
Why can't we outsource the inconsistency checking to VLM?
To justify the necessity of our design, we also try to prompt VLM with two screens and ask it to report the inconsistencies. We use the following prompt:
# We find the VLM has severe hallucination issues:
(1) It reports non-existent inconsistencies
(2) It refers to the wrong widget IDs
(3) It matches two widgets from the same screen.
# We find the VLM has severe hallucination issues:
(1) It reports non-existent inconsistencies
(2) It refers to the wrong widget IDs
(3) It matches two widgets from the same screen.
Google Sites
Report abuse