Phase 1

Prototype

In evaluation phase 1, participates were asked to freely navigate the prototype for 5 mins, then complete a short survey and a oral feedback.

Survey

In the survey, participants are asked to rate their experiences on a scale of 1 to 7, with 1 being the worst and 7 being the best. The questions are as follow:

1. On 1 scale of 1-7, how would you rate your experience with this prototype?

2. On a scale of 1-7, how would you rate the aesthetic design of the website?

3. On a scale of 1-7, how easy is it to navigate the website?

4. For commenting function: On a scale of 1-7, how useful do you think this function will be in enhancing your reading and experience.

5. For chat group function: On a scale of 1-7, how useful do you think this function will be in enhancing your reading and experience.

6. For study room function: On a scale of 1-7, how useful do you think this function will be in enhancing your reading and experience?

Result & Analysis:

The score is calculated as: Σ (% of all responds × score).

Based on the survey, participants had a good overall experiences and stated that they are likely to use it if implemented in real life. Out of the 3 main functions, commenting function was rated the highest, and study room function was rated the lowest. This is consistently with our priorities in the next development phase. In the development phase, we will first focus on the development of the commenting function, and study room function will have the lowest priority.

Oral Feedback

Participants are asked to give a oral feedback upon the completion of survey. The questions are as follow:

1. Based on the survey you just filled out, please elaborate on why you gave these ratings.

2. What do you like about the website?

3. What don’t you like about the website?

4. Which function do you like the most? Why?

5. Which function do you like the least? Why?

6. Is there anything you would like to add about your experience?

Result & Analysis:

Feedback highlight: 90% of the participants liked the design and layout of the prototype.

Best liked function:

• Comment, 60%

• Saved article, 10%

• Chat room, 30%

• Study group, 20%

Least liked function:

• Chat room, 20%

• Commenting, 10%

• Study room, 20%

• Nothing, 30%

*Because some people selected multiple functions as their top/bottom choice and some people didn't specify, the % does not add up to 100%.

Main issues raised by the participants:

1. Don’t know the difference between the chat room and study room , 80%

2. Buttons/bar needs consistency, 50% (The navigation bar’s size and location keeps changing and it confuses the participants)

3. The current design of the commenting function is distracting (change to “hover highlight”), 20%

4. If there is no real time user, chat room is wired to use, 20%

Suggestions:

1. Enable a search button and a return button, 30%

2. Enable a bookmark function. Also indicate when the article is saved and for what purpose, 20%

3. Thumb up and down function, 10%

Overall, people liked the design and layout of the current prototype, so we will keep the same format when developing the real website. However, there were some issues and suggestions that were repeatedly mentioned during the evaluation. The list above concluded these issues, and we plan to make changes accordingly. Specifically, we plan to make the following changes:

1. Change the entry point and the design of the study room and chat room to further differentiate the 2 functions.

2. Change the design to make sure that navigation bars are consistent in size, layout, and location.

3. Enable a search and return button.

Due to the time constraint, we will focus on implementing the above 3 changes. If we have more time, we will consider adding features such as bookmarking, hover highlighting, and real-time user indication.

Phase 2

In phase 2, participates were asked to complete a 2 parts evolution. In the first part, participants were asked to participate in a within subject experiment to evaluate the usefulness of the website. In the second part, participants were asked to evaluate the website as in phase 1.

Part 1

Result & Analysis

Part 2

In this part of evaluation phase 2, same survey and interview questions as Phase 1 were used.

Result & Analysis

Feedback highlight: 70% of the participants liked the commenting function. They liked to have real time interaction while reading.

Main issues raised by the participants:

1. Article too small, 30%

2. "Leave the discussion room" not obvious, 30%

Overall, people liked the design and layout of the current prototype, and they liked the idea of having real time interaction while reading. Although there were still a lot of room for improvement, major issues raised in phase 1 were resolved in this prototype.