Evaluation

A Quick Recap

Final Solution

Based on our analyses, the final solution is an interactive online course, accessible on both web and mobile platforms, integrates multimedia learning, simulation games, and online forums across four structured sections.

Prototype - Week 1 "Understand Sleep Science"

Evaluation Plan

Purpose

The content to be evaluated is the instruction of week 1: Understand the Sleep Science, which mainly focuses on the introduction, basic ideas and significance of sleeping. This week, learners will try the simulation at the beginning and end of the instruction to implement strategies for managing the character's sleep. They will also watch a series of videos to understand the importance of sleep, circadian rhythms, model of sleep regulation, etc.

The issues we expect to figure out from the formative evaluation include:

What is learners' impression of our product?
Are learners able to understand the structure and content of the instruction from our product?
What part of our product is working?
What part of the product needs to be improved?
How can it be improved?

The purpose of this evaluation is to know learners’ reactions to the content of week 1. The evaluation could provide answers to the questions mentioned above and help the design team to improve the program and the design process.

Method

In our formative evaluation, we employ a usability test coupled with a brief post-session interview. This combination of methods was selected for its effectiveness in gaining insights into user interaction with our product and gathering immediate feedback on user experiences and challenges.

To illustrate, the usability test allows us to observe firsthand how users perceive and respond to our visual designs, UX writings, and navigation system, identifying any usability issues or areas of confusion. The post-session interview complements this by providing an opportunity for users to articulate their thoughts, preferences, and suggestions, thereby enhancing the validity of our findings.

In the session, we plan to conduct tests with a sample of 3 NYU students. This convenience group aligns with our target audience, enabling us to assess the efficiency of the platform's clarity and navigation among users within our intended demographic.

Tasks:
1. Navigate to the session titled “Understand the Sleep Science” and briefly describe how you would do so.

2. Find the learning content for this session. Please locate and open them.

3. Explain how the course is divided into modules.

4. Locate the section where simulation games are provided.

5. Open the community and participate in a forum discussion in week 1.

Interview questions:

What were your overall impressions of navigating through the course platform? Did you encounter any difficulties or frustrations while completing the tasks?
How easy was it for you to find your way around the course materials?
Did you encounter any difficulties in identifying the functional buttons on the website?
How do you feel about the structure of the course regarding clarity and logic?
Were you able to understand how the course was organized and what was expected of you?
Do you have any suggestions for how we could improve the clarity of the course platform?

Process

Conducting the Usability Test

Briefly introduce the purpose of the session to the participants, explaining what they will be doing and how they should approach the tasks.
Have participants interact with the product. Observers should note any difficulties or hesitations as users engage with the interface. This might involve navigating through modules, using specific features, or interpreting icons and symbols.
Collect data on how participants use our design, noting any usability issues they encounter and their overall ease of use. Use screen recording, note-taking, or video recording for later analysis.

Post-Session Interview

Conduct a brief interview with each participant. Ask open-ended questions to elicit detailed responses about their experience. Focus on understanding the user's thoughts about the visual, video, simulation, process, and effectiveness of the design, any challenges they faced, and any suggestions they might have.
Record these interviews for thorough post-session analysis, ensuring all details are captured accurately.

Analysis (Sangin, 2018)

Collate Data from different participants (use pseudonyms) into 2 sections in an Affinity map:

- Observation of the usability test
- Interview

Analyze and group data in each section into:

- Visual designs
  - Hierarchy
  - Icons & Symbols
- UX writing
  - Instruction
  - Others
- Navigation
  - Structure
  - Menus
  - Links and buttons
  - Multimedia integration
  - Help and support

Aggregate data in each section

- Aggregate to “When, to Whom, what, the why, and the how of the problem” to find common themes
- Interpret insights from the aggregated data

Compare the two sections to cross-check & integrate insights

Report

- The final data (visualized if needed) and insights will be reported in a PDF format.

Evaluation Report

Summary

This formative evaluation report is intended for the course designers and focuses on insights derived from usability testing and observations combined with post-session interviews.

Major findings

Users found the platform had an intuitive user interface that facilitated easy navigation to various course sessions.
The course's content was well-organized and logically structured.
However, participants faced challenges with the community forums due to inadequate notification systems and a confusing layout, and some struggled to locate the simulation games section because it was tucked away under generalized headings.

Recommendations

The course design should be refined to improve user interaction and learning efficiency.
Enhancing the community forums with better notification systems and a more navigable layout will make it easier for users to engage in discussions.
The simulation games section should be made more accessible through clearer labeling or by placing it in a dedicated section.
Providing a clearer roadmap at the beginning of the course and offering more precise instructions for tasks and assessments will further enhance the educational experience.

Purpose

The content to be evaluated is the instruction of week 1: Understand the Sleep Science, which mainly focuses on the introduction, basic ideas, and significance of sleeping. This week, learners will try the simulation at the beginning and end of the instruction to implement strategies for managing the character's sleep. They will also watch a series of videos to understand the importance of sleep, circadian rhythms, models of sleep regulation, etc.

The issues we expect to figure out from the formative evaluation include:

What is learners' impression of our product?
Are learners able to understand the structure and content of the instruction from our product?
What part of our product is working?
What part of the product needs to be improved?
How can it be improved?

Methodology

Tasks

1. Navigate to the session titled “Understand the Sleep Science” and briefly describe how you would do so.

2. Find the learning content for this session. Please locate and open them.

3. Explain how the course is divided into modules.

4. Locate the section where simulation games are provided.

5. Open the community and participate in a forum discussion in week 1.

Interview questions

What were your overall impressions of navigating through the course platform? Did you encounter any difficulties or frustrations while completing the tasks?
How easy was it for you to find your way around the course materials?
Did you encounter any difficulties in identifying the functional buttons on the website?
How do you feel about the structure of the course regarding clarity and logic?
Were you able to understand how the course was organized and what was expected of you?
Do you have any suggestions for how we could improve the clarity of the course platform?

Results

Observation

Users were able to smoothly navigate to the "Understand the Sleep Science" session. This was achieved through an intuitive user interface that clearly labeled sections and provided an easy-to-follow path directly from the home screen.
All participants were able to find and open the learning content without assistance, suggesting that our content organization meets user expectations and enhances the learning experience.
Some participants regarded the course's logical modular division, consistent content and time allocation per module, and its progressive difficulty level, making it a coherent and sequential learning experience.
Some participants found the process of participating in the community forums was not convenient. They found it particularly challenging to track ongoing discussions due to a lack of effective notification systems and a little bit confusing thread layout.
Some participants struggled to find the simulation games section due to its placement under generalized headings.

Interview

Participants found the navigation relatively straightforward and intuitive.
Most participants found it easy to locate course materials, especially through the use of clear labels and organized sections.
Participants generally had no trouble identifying functional buttons on the website.
Participants had mixed opinions on the clarity and logic of the course structure: some participants felt the course structure was clear and logical, while others found it somewhat confusing and suggested improvements in organizing content.
Some participants expressed uncertainty about the sequence of topics and suggested providing a clearer roadmap at the beginning of the course.
Some participants suggested offering clearer instructions for completing tasks and assessments.

Conclusions and recommendations

Learners generally found the navigation on the platform straightforward and intuitive with clear labels and organized sections. The modular division of content and consistent time allocation per module were highlighted as strengths. However, they encountered challenges with the community forums, where they struggled to track ongoing discussions due to ineffective notification systems and confusing thread layouts. Additionally, locating the simulation games section created difficulty for some users, since it was placed under generalized headings.

To address the challenges identified, it is recommended to enhance the functionality of the community forums by implementing effective notification systems and improving the layout for easier navigation of discussions. Revisiting the placement of the simulation games section and providing clearer labels or a separate section could help users locate it more easily. Furthermore, improvements in organizing content and providing a clearer roadmap at the beginning of the course can enhance user experience. Lastly, offering clearer instructions for completing tasks and assessments would further support users in their learning journey.

Design Process Model

Design phases

Stage 1: Research

Stage 2: Identify

Stage 3: Design

Stage 4: Iteration

fff

Stage 5: Launch and Post-Launch

Explanation

Before design begins, thorough research is conducted to understand project goals, audience needs, and contextual factors. This phase involves needs, learner, context, and content analysis, ensuring alignment with stakeholder expectations.

The identification stage clarifies project objectives and parameters. Learning goals and objectives are defined, the target audience is refined, and design implications are considered, ensuring strategic alignment with project goals.

Building on research insights, the design stage creates a cohesive learning experience. A logic model guides the design process, while a compelling narrative and interactive activities engage learners. Learning materials and assessment strategies are developed to support learning.

In the iterative phase, designs are refined through development and evaluation cycles. Prototypes are created and tested, from low to high fidelity, with formative evaluations conducted to assess effectiveness. Stakeholders analyze results and provide feedback, informing iterative improvements.

The final stage sees the project ready for implementation. Performance is monitored, user feedback is collected to guide ongoing improvements, and features are continuously enhanced based on user insights. This iterative approach ensures the project remains responsive to user needs and organizational objectives post-launch.

Rationale

The rationale behind this model is that we want to use a structured and iterative approach to ensure that the outcome is responsive to stakeholder needs and comprehensive enough to project goals.

By beginning with thorough research to understand the context, audience, and objectives, we want to establish a solid foundation for subsequent decision-making.

Each time before stepping into a further stage, requiring stakeholder approval before progression allows alignment with organizational objectives and mitigates the risk of diverging from project goals.

Furthermore, the iterative nature of the model, particularly in the development and evaluation stages, facilitates continuous refinement of our design, based on the test data and user feedback.

In retrospect, this framework systematically met the needs of both learners and stakeholders, demonstrating high adaptability.

Project Management Strategy

To ensure successful project completion, we follow a comprehensive approach that involves different stages.

Firstly, we start by identifying goals through stakeholder meetings. During this phase, we collect expectations, define the project scope, outline constraints, and establish success criteria. This helps us lay a solid foundation for our project.

Next, we move into the Planning and Prioritizing phase. Here, we assign tasks, set timelines, and allocate resources. Prioritization is crucial, and we make sure to give priority to important and urgent tasks. We also rely on documentation to provide clarity and accountability throughout this phase.

Then, we move into the Executing phase. During this phase, we focus on taking action. Regular updates keep stakeholders informed of our progress, and we ensure that any issues that arise are addressed promptly. We remain flexible and adjust our workloads as needed to maintain efficiency.

As we approach completion, we enter the Review stage. Here, we assess our progress against our objectives and identify any deviations. Asynchronous comments provide a platform for feedback, enriching our understanding and guiding strategic decisions.

Adaptability is central to our approach, and we rely on the Revise stage to make adjustments based on feedback and changing circumstances. Regular updates and diligent issue resolution guide us through this iterative process.

Finally, we reach the Deliver stage, where we present our completed project deliverables to stakeholders. This marks the culmination of our efforts, a moment to celebrate milestones achieved and challenges overcome.

Communication Tools

Reflections

During this evaluation phrase, we all learned a lot. These are our reflections:

References

Sangin, M. (2018). Observing the player experience: The art and craft of observing and documenting Games User Research. Games User Research, Anders Drachen, Pejman Mirza-Babaei, and Lennart Nacke (Eds.). Oxford University Press, Oxford, 175-188.

Page updated

Google Sites

Report abuse