Usability Testing: App Lockout

Executive Summary

This case study showcases an iterative usability evaluation I led for a new App Lockout feature, critical for security requirements within a commercial delivery driver application. The research progressed from gathering initial user feedback on paper-based screen concepts to conducting interactive testing on a live staging environment. This phased approach uncovered several significant usability issues, with the most critical being widespread user confusion regarding PIN requirements, which severely impacted task success. The findings directly informed high-priority, actionable recommendations for design revisions aimed at improving clarity, reducing driver friction, and ensuring successful feature adoption.

 Goal

The research goal was to proactively identify potential usability issues with a newly implemented App lockout screen, introduced to meet security requirements. The aim was to ensure a seamless and intuitive experience for our drivers from the outset. 

My Role

In my Role as User Researcher I led this usability test for the App Lockout feature, collaborating closely with a research colleague. My responsibilities encompassed the end-to-end research process, including:

• Co-designing and executing the multi-phase research strategy.

• Orchestrating all onsite research logistics, encompassing multi-location trip planning, participant coordination in alignment with depot management, and ensuring the smooth execution of all testing sessions.

• Leading all onsite user testing sessions across two distinct phases:

  • Initial feedback gathering using printed screens (Leipzig), due to feature unavailability in staging environment.

  • Interactive usability testing on a live staging environment (Cottbus), allowing for observation of real-time user behavior.

• Managing essential technical setup, which involved ensuring feature enablement on staging environments and verifying device hardware functionality.

• Analyzing both quantitative and qualitative data, collaborating with a Data Analyst to uncover deeper insights from the collected data.

• Developing and presenting actionable recommendations derived from the research findings to product and design teams, influencing future iterations.

The Challenge

Bettermile was introducing a new App Lockout feature to its delivery driver application. This feature was critical for meeting security requirements by preventing unauthorized app access after a period of inactivity. However, for our delivery drivers who operate in fast-paced, time-sensitive environments, any new security measure needed to be implemented without causing undue friction, confusion, or delays in their critical daily workflows.

The primary challenge was to ensure that drivers could:

1. Understand and manage the pre-lockout warnings effectively.

2. Intuitively and quickly unlock the app when it did lock.

3. Clearly comprehend which PIN was required for the process, especially given they use multiple PINs (device vs. application-specific).

Anticipating potential usability hurdles with these new screens was crucial. Therefore, this usability research was initiated to proactively identify pain points and gather direct user feedback before a wider rollout, aiming to refine the feature for optimal driver experience and security compliance.

Methodology

Participants: Conducted with 37 delivery drivers across two German depots (Leipzig & Cottbus) in March and April 2025

Method: Due to the feature not being available in a testable digital environment at this stage, printed paper versions of key app lockout screens were presented to drivers. Their understanding of the flow and intended interactions were gathered using printed out paper version of the App screens. This involved showing the static screens and asking three targeted questions for each to efficiently collect data. 

In a second iterative approach interactive usability testing tasks were administered using the live staging environment on company-provided Android devices. 

Task tested: 

• Preventing Lockout: Users were presented with an on-screen alert with a visible countdown timer indicating an impending temporary app lock. They needed to find how to dismiss or pause this alert.

• Unlocking the App: After the app simulated a lockout, users were shown a screen requiring an action (tapping a specific text label) to proceed to PIN entry to unlock the App.

• PIN Entry: Users were directed to a screen to enter a PIN using a numerical keypad to regain app access.

Metrics: Key metrics included task success rates, average attempts per user, and qualitative feedback on user understanding and interaction patterns.

Key Findings & Insights

Task Preventing Lockout

Task Unlocking the App

Task PIN Selection

Actionable Recommendations

The iterative nature of the testing, especially the deeper insights from the Cottbus phase, led to the following key recommendations:

• Pre-Lockout Alert: Fundamentally redesign the alert for unambiguous interaction, ensuring the dismiss/snooze action is immediately obvious and accessible.

• Unlock Mechanism: Beyond text changes, explore significant UI redesigns for the unlock screen to create a clear, prominent, and intuitive primary action for initiating PIN entry. Test multiple design variations.

• PIN Entry Clarity: Implement explicit, unmistakable instructions on the PIN entry screen e.g., 'Enter your Device PIN'. Visually differentiate this step if possible and provide contextual help.

Learnings & Reflections

This project was a significant learning experience in adapting research methodologies based on feature availability and development timelines. It highlighted the distinct values of different fidelity testing.

Value of Low-Fidelity Testing: Using printed paper screens in Leipzig for the initial testing proved to be a highly effective and efficient method for gathering quick, early feedback on core concepts and potential major misunderstandings when an interactive prototype wasn't available. It helped identify key areas of concern at a very low cost.

Value of a phased testing approach: Initial feedback from paper-based visuals in Leipzig effectively flagged core usability concerns, which were largely confirmed by the subsequent interactive testing in Cottbus. While the interactive phase was invaluable for observing nuanced user behavior, the consistency of major findings across both fidelities provided robust validation of the primary usability challenges.

Adaptability as a Researcher: This project reinforced the importance of being flexible with research plans and choosing methods that best fit the project stage, constraints, and research questions.