This case study showcases an iterative usability evaluation I led for a new App Lockout feature, critical for security requirements within a commercial delivery driver application. The research progressed from gathering initial user feedback on paper-based screen concepts to conducting interactive testing on a live staging environment. This phased approach uncovered several significant usability issues, with the most critical being widespread user confusion regarding PIN requirements, which severely impacted task success. The findings directly informed high-priority, actionable recommendations for design revisions aimed at improving clarity, reducing driver friction, and ensuring successful feature adoption.
Co-designing and executing the multi-phase research strategy.
Orchestrating all onsite research logistics, encompassing multi-location trip planning, participant coordination in alignment with depot management, and ensuring the smooth execution of all testing sessions.
Leading all onsite user testing sessions across two distinct phases:
Initial feedback gathering using printed screens (Leipzig), due to feature unavailability in staging environment.
Interactive usability testing on a live staging environment (Cottbus), allowing for observation of real-time user behavior.
Managing essential technical setup, which involved ensuring feature enablement on staging environments and verifying device hardware functionality.
Analyzing both quantitative and qualitative data, collaborating with a Data Analyst to uncover deeper insights from the collected data.
Developing and presenting actionable recommendations derived from the research findings to product and design teams, influencing future iterations.
Bettermile was introducing a new App Lockout feature to its delivery driver application. This feature was critical for meeting security requirements by preventing unauthorized app access after a period of inactivity. However, for our delivery drivers who operate in fast-paced, time-sensitive environments, any new security measure needed to be implemented without causing undue friction, confusion, or delays in their critical daily workflows.
The primary challenge was to ensure that drivers could:
Understand and manage the pre-lockout warnings effectively.
Intuitively and quickly unlock the app when it did lock.
Clearly comprehend which PIN was required for the process, especially given they use multiple PINs (device vs. application-specific).
Anticipating potential usability hurdles with these new screens was crucial. Therefore, this usability research was initiated to proactively identify pain points and gather direct user feedback before a wider rollout, aiming to refine the feature for optimal driver experience and security compliance.
Participants: Conducted with 37 delivery drivers across two German depots (Leipzig & Cottbus) in March and April 2025
Method: Due to the feature not being available in a testable digital environment at this stage, printed paper versions of key app lockout screens were presented to drivers. Their understanding of the flow and intended interactions were gathered using printed out paper version of the App screens. This involved showing the static screens and asking three targeted questions for each to efficiently collect data.
In a second iterative approach interactive usability testing tasks were administered using the live staging environment on company-provided Android devices.
Task tested:
Preventing Lockout: Users were presented with an on-screen alert with a visible countdown timer indicating an impending temporary app lock. They needed to find how to dismiss or pause this alert.
Unlocking the App: After the app simulated a lockout, users were shown a screen requiring an action (tapping a specific text label) to proceed to PIN entry to unlock the App.
PIN Entry: Users were directed to a screen to enter a PIN using a numerical keypad to regain app access.
Metrics: Key metrics included task success rates, average attempts per user, and qualitative feedback on user understanding and interaction patterns.
Initial Findings - First depot visit Leipzig
Initial tests with 14 in Leipzig showed that 57.1% successfully found the button to stop the lockout, while 42.9% struggled with this initial interaction.
Follow-up Findings & Deeper Insights - Second depot visit Cottbus
More focused observation in Cottbus with a smaller group confirmed this challenge: 60% understood the required action, but 40% did not, indicating the issue persisted and required design attention.
Initial Findings - First depot visit Leipzig
In Leipzig, the correct text button to unlock the app was found by only 29.4% of drivers on their first attempt. Common errors included attempting to press the 'Lock Icon' (29.4%) or 'Logout' (23.5%)
Follow-up Findings - Cottbus, with revised translation
Learning from the Leipzig visit, a revised German translation for the unlock prompt was tested in Cottbus. This iteration led to a modest increase in first-attempt success to 37%. However, eventual success (53%) still often required multiple attempts (average 1.42), with users continuing to explore other UI elements like the lock icon or swipe gestures first.
Initial Findings - First depot visit Leipzig
The initial Leipzig tests already highlighted significant confusion regarding PIN selection: only 33.3% of users correctly identified the Device PIN as the one required, with the majority (66.7%) incorrectly selecting their application-specific PIN
Follow-up Findings - Cottbus
Alarmingly, this critical issue was even more pronounced in the Cottbus tests. Only 17% selected the correct Device PIN, while 83% opted for the incorrect application-specific PIN. This regression underscored the severity of the problem.
Pre-Lockout Alert: Fundamentally redesign the alert for unambiguous interaction, ensuring the dismiss/snooze action is immediately obvious and accessible.
Unlock Mechanism: Beyond text changes, explore significant UI redesigns for the unlock screen to create a clear, prominent, and intuitive primary action for initiating PIN entry. Test multiple design variations.
PIN Entry Clarity: Implement explicit, unmistakable instructions on the PIN entry screen e.g., 'Enter your Device PIN'. Visually differentiate this step if possible and provide contextual help.
Value of Low-Fidelity Testing: Using printed paper screens in Leipzig for the initial testing proved to be a highly effective and efficient method for gathering quick, early feedback on core concepts and potential major misunderstandings when an interactive prototype wasn't available. It helped identify key areas of concern at a very low cost.
Value of a phased testing approach: Initial feedback from paper-based visuals in Leipzig effectively flagged core usability concerns, which were largely confirmed by the subsequent interactive testing in Cottbus. While the interactive phase was invaluable for observing nuanced user behavior, the consistency of major findings across both fidelities provided robust validation of the primary usability challenges.
Adaptability as a Researcher: This project reinforced the importance of being flexible with research plans and choosing methods that best fit the project stage, constraints, and research questions.