Comprehensive Evaluation Plan
Comprehensive Evaluation Plan
Embedded Files
The purpose of this evaluation plan is to understand whether our Figma prototype is clear, easy to use, and connected to our learning goal. Our prototype is designed for Grades 4–6 students. It teaches them to use the Who / Evidence / Motive routine to judge whether online information is trustworthy.
For usability
We want to know if learners can understand the prototype flow, buttons, and instructions. This is important because if students do not know how to move through the prototype, they may feel confused before they can focus on the learning content. To evaluate this, we plan to ask 2–3 peers to test the prototype as proxy learners and observe where they feel confused or stuck.
For conceptualization
We want to know if the Who / Evidence / Motive routine is clearly shown in the prototype. This routine is the core of our design, so we need to make sure learners can understand and use it. After testing, we will ask participants to explain the routine in their own words and see whether they use it in the final decision task.
For process
We want to reflect on whether our team’s development process worked smoothly. This includes our task division, Mural project plan, and team communication. We will discuss what worked well, what was difficult, and what we would change if we continued developing the prototype.
For effectiveness
We want to know whether learners can make a more thoughtful judgment after using the prototype. Our goal is not only for learners to choose an answer, but also to explain their reasoning. In the final decision task, participants will judge a new post and explain their reason using Who / Evidence / Motive.
Overall
This evaluation plan helps us check both the learner experience and the strength of our design. It also gives us a clear direction for improving the prototype in the next stage.
Conduct Evaluation
Conduct Evaluation
Evaluation Focus:
Evaluation Focus:
Learning Effectiveness
Learning Effectiveness
We want to see whether learners used more structured reasoning after going through the prototype.
We test this through a think-aloud prototype walkthrough, followed by a post-task judgment and short interview questions.
Participants
Participants
We will invite 3 peers to participate as proxy learners, since we do not currently have access to students in grades 4–6.
Using peers as proxies will still be helpful at this stage because they will be able to move through the prototype and explain their thinking clearly during the session.
Why peers?
Since target learners are not available right now, peer participants will help us test the flow, clarity, and reasoning process of the prototype.
Pre-task Judgment
Pre-task Judgment
Before showing the prototype, we will give participants two short online posts and ask whether they would trust them, ignore them, or verify them, and why.
This will give us a slightly stronger baseline of their first reactions before the learning experience, instead of relying on only one example.
What we will ask
Would you trust it, ignore it, or verify it?
Why?
Think Aloud
Think Aloud
What is Think Aloud?
Think aloud is a method where participants say their thoughts out loud while completing a task.
Instead of only looking at the final answer, we can hear how they interpret instructions, where they hesitate, and what confuses them during the process.
Why we chose it
We wanted to understand not only what participants chose, but also how they were thinking while moving through the prototype.
This was especially useful for our project because we needed to know whether learners were actually using the role cards, tool cards, and the Who / Evidence / Motive routine during the experience.
What it helped us notice
Think aloud helped us notice:
where participants paused
which instructions felt unclear
whether the cards were actually being used
whether learners mentioned source, evidence, and motive in their reasoning
What we listened for
hesitation
confusion
use of cards
use of the 3-step check
changes in reasoning
Post-task Judgment
Post-task Judgment
Post-task Judgment
After going through the prototype, participants will be shown a new online post and asked whether they would trust it, ignore it, or verify it, and why.
This step will help us see whether their reasoning becomes more structured after the learning experience.
What we will ask
Would you trust it, ignore it, or verify it?
Why?
What we will look for
clearer mention of source
clearer mention of evidence
clearer mention of motive
Follow-up Interview
Follow-up Interview
Follow-up Interview
After the post-task judgment, we will ask a few short follow-up questions to better understand how participants experienced the prototype.
This step will help us learn what felt helpful, what felt confusing, and whether the routine seemed useful to them.
Questions
What helped you most when making your decision?
What was confusing?
Did the role card or tool cards affect the way you thought?
Did the Who / Evidence / Motive routine feel useful?
Why interview?
These short questions will help us understand how participants interpreted the prototype, not just what answer they chose.
Analyze Results
Analyze Results
How We Analyzed the Results
How We Analyzed the Results
Step 1. Gather All Data
Step 1. Gather All Data
What we will do
We will collect all of the data from the evaluation, including pre-task judgments, think-aloud comments, observation notes, post-task judgments, and short interview responses.
Why it matters
This will help us keep all of our evidence in one place so the team can review it together.
Step 2. Highlight Key Moments
Step 2. Highlight Key Moments
What we will do
We will go through the notes and mark moments where participants paused, seemed confused, changed their mind, or clearly used source, evidence, or motive in their reasoning.
Why it matters
This will help us identify which parts of the experience stand out and which comments appear more than once.
Step 3. Group Findings into Themes
Step 3. Group Findings into Themes
What we will do
We will group our notes into a few themes, such as card clarity, pacing, reasoning quality, and connection to the three-step check.
Why it matters
This will help us see patterns more clearly instead of treating every comment as a separate issue.
Step 4. Compare Pre/Post Reasoning
Step 4. Compare Pre/Post Reasoning
What we will do
We will compare participants’ reasoning before and after the prototype, not only to see whether their answer changed, but also whether their explanation became more structured.
Why it matters
This is the main step for judging learning effectiveness, because our focus is on whether learners begin to use source, evidence, and motive more clearly.
Step 5. Identify Priority Issues
Step 5. Identify Priority Issues
What we will do
We will decide which issues appeared most often, which ones seem most harmful to learner understanding, and which ones are most important to revise first.
Why it matters
This will help us focus on the most important improvements instead of trying to fix everything at once.
Step 6. Turn Findings into Revisions
Step 6. Turn Findings into Revisions
What we will do
We will translate the findings into concrete next steps, such as shortening card text, clarifying instructions, or improving transitions between screens.
Why it matters
This ensures that the evaluation leads to actual design changes, not just general observations.
2
2
Page updated
Report abuse