WebTestPilot

RA-e1: What's the difference between WebTestPilot and AUITestAgent, Trident?

RB-e1: How WebTestPilot converts NL test description into structured output, and how it can infer missing parts?

RB-e2: How symbolization works, how custom symbols are extracted from the state?

RB-e3: How page reidentification works, how are the page similarities computed?

RB-e4: Retry on assertion failure: number of retries n=1, the setup contradicts the introduction that LLMs are stochastic. Would increasing n help decrease false positives while balancing costs?

RB-e5: Why is the execution trace defined differently in approach and evaluation?

RB-e6: Why not use exact match instead of LCS for task completion metric?

R-All: Real-world bug experiment

R-B1: Examples where WebTestPilot outperforms PinATA

Failure Sample #1: Inability to check data across states (Train booking website)

Failure Sample #2: Out of screen data (Bookstack)

Failure Sample #3: Inability to verify data between 2 different screens (Bookstack)

R-B2, R-B3: Examples of GUI Evolution

RA-e1: What's the difference between WebTestPilot and AUITestAgent, Trident?

AUITestAgent decomposes test descriptions into atomic verification points and compares them with execution logs. Trident uses a detector agent that functions similarily to PinATA's verifier agent. In contrast to WebTestPilot, they:

Extract textual evidence from logs or other sources rather than symbolic information.
Following approaches like PinATA, it relies on verification that produces natural-language judgments of correctness.

These design choices have several implications:

The system is not instructed to infer cross-state or implicit requirements, which WebTestPilot explicitly models.
Unlike symbolic reasoning, natural language verification is vulnerable to hallucination and stability issues, as shown in our motivating study.
Unlike assertion code, natural language judgements cannot be repeatedly and deterministically executed, nor are they easily interpretable by testers.

In addition, both of them focus on the Android domain, and AUITestAgent's implementation is not open-sourced.

RB-e1: How WebTestPilot converts NL test description into structured output, and how it can infer missing parts?

Given a test description: `…type "admin@admin.com" in the email field…`

During the Input Parsing stage, WebTestPilot parses the description as a (condition, action, expectation) tuple:

```json

{

“condition”: “<MISSING>” ,

“action”: “type ‘admin@admin.com’ in the email field”,

“expectation”: “<MISSING>”

}

```

It automatically infers and fills in any missing steps by considering the context of the entire test description.

```json

{

“condition”: “email field is present” ,

“action”: “type ‘admin@admin.com’ in the email field”,

“expectation”: “the text ‘admin@admin.com’ is present in email field”

}

```

RB-e2: How symbolization works, how custom symbols are extracted from the state?

Symbolization consists of 2 steps: symbol definition and symbol extraction.

Symbol definition happens alongside oracle inference. When generating assertions, the LLM prompted by WebTestPilot will also declare the symbols that will be used in the assertions. These symbols are defined as Pydantic BaseModel subclasses, specifying the symbol name (e.g., Cart, Item) and its attributes (e.g., Cart.items is a list of Item, and Item has price and quantity). The assertion code can use a predefined function extract() part of the API to create instances of these symbols, associating them with the relevant target state (e.g., a previous application state).

Symbol extraction occurs at runtime during oracle execution. When extract() is called, the LLM is prompted with the symbol schema and a screenshot of the target state. It then outputs a symbol instance populated with values derived from the state.

RB-e3: How page reidentification works, how are the page similarities computed?

RB-e4: Retry on assertion failure: number of retries n=1, the setup contradicts the introduction that LLMs are stochastic. Would increasing n help decrease false positives while balancing costs?

RB-e5: Why is the execution trace defined differently in approach and evaluation?

RB-e6: Why not use exact match instead of LCS for task completion metric?

We use LCS to filter out unrelated actions (e.g., wait) from baselines. We tried an exact match and confirmed that it is consistent with our results.

R-All: Real-world bug experiment

Attached is the list of real-world bugs run results from WebTestPilot and Pinata. WebTestPilot can detect 23 out of 24 bugs (96%), compared to 17 out of 24 bugs (71%) from Pinata.

Among the 7 bugs that Pinata was not able to detect:

1 due to missing cross-state context
2 from misinterpreting requirements
4 from PinATA verifier hallucination on dense or highly detailed pages.1 lacks access to the cross-state context to verify consistency.

WebTestPilot Real-World Bugs

R-B1: Examples where WebTestPilot outperforms PinATA

Failure Sample #1: Inability to check data across states (Train booking website)

Test case given with 1 step:

```javascript

{

"actions": [

{

"action": "Click on the date 25 november",

"expectedResult": "There are 30 trains listed in total across 2 days"

}

]

}

```

Pinata logs

=== Step 1, Action: "Click on the date 25 november" ===

Orchestrator Plan response:

- Actor: "Click on the date button labeled '2025-11-25' at the top of the page."

- Assertor: "Verify that after clicking, there are train listings shown for two different dates and that their combined total number of trains listed is exactly 30."

Actor -> success

Assertor verdict: {

"status": "fail",

"discrepancies": "Only one day's train listings are shown; there are no visible groupings or indicators for two different dates nor confirmation that there are exactly 30 trains listed in total across two days."

}

WebTestPilot logs

Abstract Page:

```

{

"name": "Train Search Results Page",

"description": "Displays train schedules and seat availability for selected dates in a tabular format.",

"layout": "<Page>...</Page>"

}

```

=== Step 1, Action: "Click on the date 25 november" ===

Proposed Actions: `click(target_description="2025-11-25 date button")`

Closest state: Train Search Results Page

Is same logical page: true

Generate Post-condition:

```python

def postcondition(session: Session):

from pydantic import BaseModel, Field

from typing import List

# Define schema for a train schedule row

class TrainSchedule(BaseModel):

train: str = Field(..., description="Train identifier")

from_station: str = Field(..., description="Departure station")

to_station: str = Field(..., description="Arrival station")

depart: str = Field(..., description="Departure time")

arrive: str = Field(..., description="Arrival time")

duration: str = Field(..., description="Travel duration")

second_class: str = Field(..., description="2nd class seat availability")

first_class: str = Field(..., description="1st class seat availability")

business: str = Field(..., description="Business class seat availability")

note: str = Field(..., description="Additional notes")

class TrainResults(BaseModel):

schedules: List[TrainSchedule] = Field(..., description="List of train schedules displayed in the table")

# Extract train results for the currently selected date (after clicking 2025-11-25)

current_results = session.history[-1].extract("get all train schedules in the table", schema=TrainResults).schedules

# Extract train results for the previous date (2025-11-24, before the click)

previous_results = session.history[0].extract("get all train schedules in the table", schema=TrainResults).schedules

# Assert that the total number of trains across both days is 30

total_trains = len(current_results) + len(previous_results)

assert total_trains == 30, f"Expected 30 trains across two days, but found {total_trains}."

# Assert that both days have at least one train listed

assert len(current_results) > 0, "No trains listed for the selected date (2025-11-25)."

assert len(previous_results) > 0, "No trains listed for the previous date (2025-11-24)."

```

ExtractFromState: "get all train schedules in the table" -> 15 trains

POST-CONDITION PASSED!

Failure Sample #2: Out of screen data (Bookstack)

Test case given with 2 steps:

```javascript

{

"actions": [

{

"action": "Click on my profile icon",

"expectedResult": "A dropdown with profile options with \"My Favourites\" appears"

{

"action": "Click on My Favourites option",

"expectedResult": "Shows 10 favourited items"

}

]

}

```

Pinata logs

=== Step 1, Action: "Click on my profile icon" ===

Orchestrator Plan response:

- Actor: "Click on the profile icon (user avatar with label 'Admin') in the top right corner of the page."

- Assertor: "Verify that a dropdown menu appears below or near the profile icon and contains an option labeled 'My Favourites'."

Actor -> success

Assertor -> success

=== Step 2, Action: "Click on My Favourites option" ===

Orchestrator Plan response:

- Actor: "Click on the 'My Favourites' option in the profile dropdown menu."

- Assertor: "Verify that a section or page appears showing exactly 10 favourited items."

Actor -> success

Assertor verdict: {

"status": "fail",

"discrepancies": "Only seven favourited items are shown instead of the required ten. No evidence of more items being present within this view."

}

WebTestPilot logs

Abstract Page:

```

{

"name": "User Dashboard",

"description": "Displays recent activity, favorites, and updates in a clean, card-based layout.",

"layout": "<Page>...</Page>"

}

```

=== Step 1, Action: "Click on my profile icon" ===

Proposed Actions: `click(target_description="profile icon in the top right corner")`

Closest state: User Dashboard

Is same logical page: true

Abstract Page:

```

{

"name": "Books Library Overview Page",

"description": "Displays book cards, navigation, and actions; visually clean with sidebar and top bar.",

"layout": "<Page>...</Page>"

}

```

... Post condition checking ... -> Pass

=== Step 2, Action: "Click on My Favourites option" ===

Proposed Actions: `click(target_description="View All link under My Most Viewed Favourites")`

Closest state: User Dashboard

Is same logical page: false

Abstract Page:

```

{

"name": "Favourites List Page",

"description": "Displays user’s favorite books, chapters, and shelves in a clean, centered list.",

"layout": "<Page>...</Page>"

}

```

Generate Post-condition:

```python

def postcondition(session: Session):

from pydantic import BaseModel, Field

from typing import List

class FavouriteItem(BaseModel):

title: str = Field(..., description="Title of the favourited item")

description: str = Field(..., description="Description of the favourited item")

class FavouritesList(BaseModel):

items: List[FavouriteItem] = Field(..., description="List of favourited items")

# Extract the favourites list from the current state

favourites = session.history[-1].extract("get all favourited items", schema=FavouritesList)

# Assert that there are exactly 10 favourited items shown

assert len(favourites.items) == 10, f"Expected 10 favourited items, but found {len(favourites.items)}."

```

ExtractFromState: "get all favourited items"

```

{

"schema": {

"items": [

{

"title": "Book1",

"description": "Description"

... 9 more elements

]

}

```

POST-CONDITION PASSED!

Failure Sample #3: Inability to verify data between 2 different screens (Bookstack)

Test case given with 3 steps:

```javascript

{

"actions": [

{

"action": "Click on the \"Books\" navigation item",

"expectedResult": "Books listing page appears"

{

"action": "Click on my profile icon",

"expectedResult": "A dropdown with profile options with \"My Favourites\" appears"

{

"action": "Click on My Favourites option",

"expectedResult": "Favourite items are consistent with ones on home page"

}

]

}

```

Pinata logs

=== Step 1, Action: "Click on the \"Books\" navigation item" ===

Orchestrator Plan response:

- Actor: "Click on the 'Books' navigation item in the top blue navigation bar."

- Assertor: "Verify that after clicking, you are on the Books listing page by checking for a heading or title indicating 'Books' and a list of books displayed."

Actor -> success

Assertor -> success

=== Step 2, Action: "Click on my profile icon" ===

Orchestrator Plan response:

- Actor: "Click on the 'Admin' profile icon in the top right corner of the navigation bar."

- Assertor: "Verify that a dropdown menu appears below the profile icon and that one of its options is labeled 'My Favourites'."

Actor -> success

Assertor -> success

=== Step 3, Action: "Click on My Favourites option" ===

Orchestrator Plan response:

- Actor: "Click on the 'My Favourites' option in the profile dropdown menu."

- Assertor: "Verify that all item titles listed as favourites match those shown in any favourites-related widget or section on the homepage (such as Most Viewed Favourites)."

Actor -> success

Assertor verdict: {

"status": "unknown",

"discrepancies": "Unable to verify assertion due to lack of visibility of the homepage or its favourites-related widgets/sections in this screenshot."

}

WebTestPilot logs

Abstract Page:

```

{

"name": "User Dashboard Page",

"description": "Displays recent activity, favorites, and updates in a clean, card-based layout.",

"layout": "<Page>...</Page>"

}

```

=== Step 1, Action: "Click on the \"Books\" navigation item" ===

Proposed Actions: `click(target_description="Books navigation item in the top right navigation bar")`

Closest state: User Dashboard Page

Is same logical page: false

Abstract Page:

```

{

"name": "Books Library Overview Page",

"description": "Displays book cards, navigation, and actions; visually clean with sidebar and top bar.",

"layout": "<Page>...</Page>"

}

```

... Post condition checking ... -> Pass

=== Step 2, Action: "Click on my profile icon" ===

Proposed Actions: ```click(target_description="profile icon in the top right corner")```

Closest state: Books Library Overview Page

Is same logical page: false

Abstract Page:

```

{

"name": "Books Overview Page",

"description": "Displays book cards, sidebar lists, and user actions in a clean, card-based layout.",

"layout": "<Page>...</Page>"

}

```

... Post condition checking ... -> Pass

=== Step 3, Action: "Click on My Favourites option" ===

Proposed Actions: ```click(target_description="My Favourites option in the dropdown menu at the top right")```

Closest state: User Dashboard Page

Is same logical page: false

Abstract Page:

```

{

"name": "Favorites List Page",

"description": "Displays user’s favorite books, chapters, pages, and shelves in a clean layout.",

"layout": "<Page>...</Page>"

}

```

Generate Post-condition:

```python

def postcondition(session: Session):

from typing import List

from pydantic import BaseModel, Field

# Define schema for a favourite item

class FavouriteItem(BaseModel):

title: str = Field(..., description="Title of the favourited item")

description: str = Field(..., description="Description of the favourited item")

class FavouritesList(BaseModel):

items: List[FavouriteItem] = Field(default_factory=list, description="List of favourited items")

# Extract favourites from the dashboard (homepage)

dashboard_favourites = session.history[0].extract(

"extract all items in the 'My Most Viewed Favourites' card, including their titles and descriptions",

schema=FavouritesList

).items

# Extract favourites from the current favourites page

favourites_page_favourites = session.history[-1].extract(

"extract all items in the favourites list, including their titles and descriptions",

schema=FavouritesList

).items

# Compare titles (order-insensitive)

dashboard_titles = set(item.title for item in dashboard_favourites)

favourites_page_titles = set(item.title for item in favourites_page_favourites)

assert dashboard_titles == favourites_page_titles, (

f"Favourited item titles on the homepage ({dashboard_titles}) do not match those on the Favourites page ({favourites_page_titles})."

)

```

POST-CONDITION PASSED!

R-B2, R-B3: Examples of GUI Evolution

Amazon: before (left) and after (right)

USPS: before (left) and after (right)

Bookstack, Add GUI Elements: before (left) and after (right)

Indico, Add GUI Elements: before (left) and after (right)

Google Sites

Report abuse