Introduction

E2E Web Testing from Natural Language is promising and developing rapidly, but faces the challenge of unreliability and verifying app behaviour independently.

We introduce WebTestPilot, an LLM-based agent designed to address these challenges. It combines:

1. A symbolization layer that models all page states in the test, capturing data, temporal, and causal dependencies as contextual symbols.

2. A domain-specific language (DSL) that translates natural language requirements into formal pre- and post-condition assertions on these symbols. 

We also contributed a benchmark of bug-injected web apps for evaluating NL-to-E2E testing.

Demo