Embedded Files
Evaluation Methods
To evaluate StudyNest, our team conducted two rounds of usability testing using a Wizard of Oz approach with low-fidelity and high-fidelity prototypes. We also performed a heuristic evaluation using Nielsen's 10 usability heuristics to identify strengths and potential usability issues.
Our goal was to understand whether users could successfully navigate the platform, use the AI Tutor, manage study plans, and understand how the adaptive learning system worked.
Participants
Participants were undergraduate students enrolled in INFO 360, who closely matched our target audience of university students who use AI tools for academic support.
The first usability test was conducted with students from Group 8, while the second usability test was conducted with students from Group 2.
Tasks Performed
Participants were asked to complete realistic learning-related tasks, including:
Using the AI Tutor to ask academic questions
Testing guided hints and support levels
Creating study goals and deadlines
Adding study sessions to the planner
Exploring progress tracking and achievement features
Discussing Canvas integration and personalization features
These tasks allowed us to evaluate both the learning experience and the overall usability of the platform.
What Worked Well
Overall, participants responded positively to StudyNest's core concept of learning through guidance rather than receiving immediate answers.
Users particularly liked:
The AI Tutor's step-by-step guidance
Adaptive support levels
Progress tracking and mastery visualization
Study planning and goal-setting features
Achievement and motivation systems
Many participants stated that the platform felt more educational than traditional AI chatbots because it encouraged them to think through problems independently.
Challenges and User Feedback
Although users appreciated the concept, several usability issues came up during testing.
Participants were sometimes confused about:
Whether StudyNest connected directly to Canvas
Where academic data and progress information came from
How the hint system worked
Whether the AI would eventually provide direct answers after multiple attempts
Users also requested:
More personalized recommendations
Better explanations of AI decision-making
More specific practice questions
Additional social and collaborative features
These findings showed that transparency and personalization were important areas for improvement.
Figure 3. User journey map showing key pain points and opportunities identified during usability testing.
Heuristic Evaluation
Our heuristic evaluation found that StudyNest performed well in several areas. The design maintained consistent navigation, visual styling, and terminology across screens, which supported usability and learnability. Progress indicators, study streaks, mastery trackers, and onboarding steps helped users understand their current status. Users could also move easily between the Dashboard, AI Tutor, Planner, and Progress pages.
However, we identified opportunities to improve, including clearer confirmation messages after syncing or creating study sessions, better error messages for invalid outputs, clearer explanations of AI support levels, and more help documentation for first-time users.
Key Lessons Learned:
The evaluation process confirmed that students value AI tools that support understanding rather than simply providing answers. At the same time, users want transparency about how AI operates, where academic data comes from, and how adaptive support decisions are made.
These findings helped us refine StudyNest into a more understandable, supportive, and student-centered learning platform.
Page updated
Report abuse