With this Evaluation Plan, we are able to answer all the important questions:

Evaluate issues 

for the core activity, the Spreadsheet related to short-term outcomes (logic model)

• • • Issue 1: How easy is it for students to use the materials?  

      • Short-term outcome 1 

      • goal: Students know where to find the NYU Brooklyn resources they need to do the things they want to do.

        • Objective: Students will be able to navigate and organize the resources that they need.

    • Issue 2: To what extent do students feel beneficial/engaged/motivated to interact with the learning materials we’ve designed for them?

      • Short-term outcome 2/3

      • goal: Students know what sorts of resources NYU Brooklyn has to offer.

      • goal: Students understand the value of interdisciplinary collaboration.

        • Objective: Students will be able to examine and evaluate the interdisciplinary nature of their own learning experiences.

Evaluate issues 

for the core activity, the Spreadsheet related to short-term outcomes (logic model)

• • • Issue 1: How easy is it for students to use the materials?  

      • Short-term outcome 1 

      • goal: Students know where to find the NYU Brooklyn resources they need to do the things they want to do.

        • Objective: Students will be able to navigate and organize the resources that they need.

    • Issue 2: To what extent do students feel beneficial/engaged/motivated to interact with the learning materials we’ve designed for them?

      • Short-term outcome 2/3

      • goal: Students know what sorts of resources NYU Brooklyn has to offer.

      • goal: Students understand the value of interdisciplinary collaboration.

        • Objective: Students will be able to examine and evaluate the interdisciplinary nature of their own learning experiences.

Audience

Participants - Students on the Brooklyn campus / potential workshop participants 

• • • N=2 students; an ECT graduate student and an IDM undergraduate student

Description of the instruction being evaluated

The main instructions of the Spreadsheet introduction have three steps:

• • 1. Introduction to Resource Categories

    2. Spreadsheet Navigation and Data Entry

    3. Resource Identification

Usability testing

• The methodology is delineated into four distinct tasks to measure user performance and better understand and empathize with our users:

  • 2. 1. 1. Account creation

          2. Spreadsheet duplication

          3. Chart fill-in

          4. Find project-related resources using a filter

Evaluation Methodology 

Associated methods and rationale

Interviews 

• • • Provide rich qualitative insights into participants' experiences, preferences, and perceptions, complementing the quantitative data collected through observational methods.

Usability Testing

• Participants are asked to perform tasks while observers watch and take notes in a user observation form, providing insights for user interactions with a spreadsheet template.

Participants 

• • • N=2 students; an ECT graduate student and an IDM undergraduate student

Instruments & Study Design

Study Design

• • • Workshop about NYU resources where they track their resources in a spreadsheet. In workshop, students will create a notion account, duplicate the spreadsheet, and add information to the spreadsheet

    • Construct: Level of satisfaction amongst users

    • Measures: Task Success Rate (completion), Error Rate (wrong clicks), Utterances (valence, keywords/quotes, locus)

    • Instruments: User Observations, Audio Recordings

Users have four tasks

• • • 1. Create a notion account 

      2. Duplicate spreadsheet template into own account (link provided to users)

      3. Provide users with two NYU resources (websites) and ask them to put relevant information into a spreadsheet.

      4. Find project-related resources using a filter.

Data Collection

• • • User observation form, recordings, e-documents, Notion website

Quantitative

• • • User observations: Evaluating success in task completion. Wrong clicks/navigation. Utterance sentiments.

Qualitative

• • • User observations: Unprompted utterances with keywords.

Data Analysis

Task Analysis

• Successful task completion rates

• Errors/deviance rates. Identify locus point of errors and make modifications.

Qualitative Analysis

• Grounded open coding of utterances (keywords)

Quantitative Analysis

• Nominally code observations and use descriptive statistics, frequencies, and co-occurrences

Evaluation Results

Analysis and method:

Task Analysis: 

• Break down the spreadsheet tasks into smaller steps

• Analyze efficiency and effectiveness of each step

• Identify areas of interest and pain points in spreadsheet interface

Qualitative Analysis: 

• Review data from observations, interview transcripts, and open-ended survey responses

• Grounded coding for data according to frustrations expressed

• Identify recurring themes, issues, and user feedback

Quantitative Analysis:

• Co-occurrence analysis: Location Marker x Error x Utterance Valence

• Use to help identify areas of interest and pain points

Findings:

1. High Task Success Rate (100%) - Spreadsheet is intuitive - Task Analysis

   1. • Fulfills short-term outcome of students being able to identify and track resources and related information

2. Participants displayed a high level of engagement interacting with learning materials - Quantitative Analysis

   1. • Achieves short-term outcome we set of students understanding the value of interdisciplinary collaboration, navigating, and organizing resources

• One participant double checked sources and modified accordingly after expressing uncertainty about putting information into proper box

• Another participant described why they sorted categories from low to high priority.

3. Participants self-reflected, strategized/critically thought while using the filters and sorting mechanics for interdisciplinary resources within spreadsheet - Qualitative Analysis

   1. • Met short-term outcome of students' ability to examine and evaluate the interdisciplinary nature of their own learning experiences.

• One participant picked out several resources she thought would benefit her, e.g., maker space, library, and VIP. She said she did similar things by herself, and it would be better if she had found a spreadsheet like that when she entered the university as a freshman.

• Another participant proposed that she’s looking for internship opportunities and likes that we mentioned Handshake.

References

Wickens, C. D., Hollands, J. G., Banbury, S., & Parasuraman, R. (2015). Engineering psychology and human performance. Psychology Press.

Preece, J., Rogers, Y., & Sharp, H. (2015). Interaction design: Beyond human-computer interaction. John Wiley & Sons.

Dix, A., Finlay, J., Abowd, G., & Beale, R. (2004). Human-computer interaction. Prentice Hall.

Kirakowski, J., & Corbett, M. (1993). Summative usability testing of software: some practical advice. In CHI'93 Conference on Human Factors in Computing Systems (pp. 23-26).

Braun, V., & Clarke, V. (2006). Using thematic analysis in psychology. Qualitative Research in Psychology, 3(2), 77-101.

Miles, M. B., Huberman, A. M., & Saldana, J. (2014). Qualitative data analysis: A methods sourcebook. Sage Publications.

Creswell, J. W., & Creswell, J. D. (2017). Research design: Qualitative, quantitative, and mixed methods approaches. Sage publications.

Patton, M. Q. (2015). Qualitative research & evaluation methods: Integrating theory and practice. Sage Publications.