Embedded Files
Keywords: GUI Testing, Software Prototyping, Design Violation Checking
Introduction
Introduction
In this project, we aim to automate the detection of design violations between GUI mock-ups and their actual implementations. Traditionally, identifying such violations has required substantial manual effort and specialized domain expertise, making it particularly challenging for modern applications with hundreds of functionalities.
To address these challenges, we introduce GUIPilot, a tool designed to detect:
(i) Screen Inconsistencies, which include widget layout violations (arrangement and positioning of GUI elements) and widget semantic violations (text content and color schemes).
(ii) Process Inconsistencies, which refer to incorrect transitions between screens.
Dataset
80 public Apps from the Google Play Store.
Each App has 2 distinct user processes.
Each process comprises an average of 8 screens.
Dataset
80 public Apps from the Google Play Store.
Each App has 2 distinct user processes.
Each process comprises an average of 8 screens.
Code
Code
Screen Inconsistency: Widget detector + Widget matching
Process Inconsistency: Visual Language Model (VLM) agent
RQ1 Simulation Study on Screen Inconsistency Detection
RQ1 Simulation Study on Screen Inconsistency Detection
Demo
Demo
A short demo on how GUIPilot can facilitate modern GUI testing.
Google Sites
Report abuse