Survey Materials

Evaluation Plan

For this simple entry survey, we measure the success as simply the thorough completion of all questions, so we can have our data to finesse the learning experience for each individual learner. Rather than simply receiving many "NAs" that tell designers little about their learners, a thorough response allows for content differentiation throughout the modules.

Sign Up Survey Objectives:

• Learners will identify and evaluate with a journal reflection 2-3 artists or works they wish to emulate and/or influence their own digital storytelling.

• Instructors will acquire digital storytelling skills of their own (such as use of apps) to compile and provide learners with a "holy grail" Google Drive resource of digital storytelling tools/methods/techniques.

• Instructors will be responsible for curating this course based on student sign up submissions.

Learning theories that support our activity include:

• Constructivism: Students utilize prior knowledge & their experiences to reflect on their digital storytelling skills. This task also utilizes metacognition as a result.

• Social Constructivism: Recognizing each student as a learning resource. There is knowledge construction through collaboration and negotiating the meaning of their roles in context to group project.

• Situativity: Students will learn and interact in groups through Flipgrid, seeing other students' own Sign Up videos.

• Culturally Relevant Pedagogy: students share artists/art they’re inspired by that can be referenced to teach them digital storytelling skills in the course

Implementation & Iteration

For this activity, we had two classmates from our Design Process for Learning class undergo the sign-up survey process and video submission via Flipgrid. Although our user testers were not within the learning experience's target demographic (12-19 years old), we value their insights as instructional designers themselves.

The most important key findings focused on the wording/logic-flow of our initial Google Form. We also received feedback for using Memoji within the sign-up survey. Although it was a strong activity for building intrinsic motivation, our user testers points out that some students may be coming in with a complete lack of digital skills and find the activity frustrating.

Some refinements to consider include:

• We have a question on the survey that asks "What digital mediums are your most interested in? List top 3." Our user testers suggested we include a list of options for students to choose from, as learners might not know what "digital mediums" are.

• Our user testers also suggested that we expand on the question asking for any prior experience. They suggested we explicitly ask them to explain what they did and what software they used for deeper, more valuable data.

• Rather than asking what social media sites do students use "daily," we might consider asking what they use "most often." After all, some students might not actually use social media daily.

• We currently ask "Do you have a limited access to the internet at home or the place where you stay?" and offer only "Yes/No" as a response. Our user testers noted that it might be helpful to add a follow-up question "If yes..." This would allow learners to list any specific help they might need.

• We should consider whether the Memoji/Flipgrid video would be stronger as a small activity within the course later on.

Ultimately, our initial understanding of "success" had lacked depth. It is not enough that students "simply fill out" the survey, but that the questions are written in a way that encourages thorough answers. Furthermore, the memoji/video activity must be structured in a way that provides some insight to learner's tech skills--not just frustrates them.

• Once time is up--even if students haven't finished making their videos--they complete the Exit Ticket, which is a self-evaluation of the experience. Questions include:

  • Write down the steps you took to create this project and the features you used.

  • How long did each step take? (This will help them manage time and understand what they’re realistically able to do in their groups based on the time frame of both deliverables--the short 1-minute video and the final product.)

  • Which features from Adobe Rush did you use?

  • What free resources would you use outside of Adobe to complete this task?

  • Are you satisfied with your project? What would you do differently with more time?

  • What would you still like to know?

  • What features of the tool did you use that you are capable of bringing back to your group?

• During Day 3, learners will reconvene with their main Digital Storytelling Teams to share-out what they learned in part 1 of the Jigsaw Activity.

Materials

Evaluation Plan

• Learners will be able to identify and use the fundamental features of one Adobe tool to edit raw digital media content.

We will know that our learners were successful in the above learning goal through the Exit Ticket, which will serve as students' self-assessments of the activity. Because of this, we want to focus our beta test on a few key questions regarding the Exit Ticket specifically:

• Exit Ticket Beta Test Questions

  • Does the logic of the Exit Ticket questions flow? Do you think there is a more intuitive order?

  • Too many? Too few? Any we should replace and/or get rid of entirely?

  • Should there be questions about the content itself? (i.e. not only about the software but on how they might implement storytelling elements)

    • Following intuition vs. following a script

Additional proof of success can be seen through instructor observation during the Adobe Rush 1-minute video challenge. Instructors can monitor learners' individual comfort with the activity and where they might be struggling and/or engaging with the tool. This can be measured by students' speed with the features, how long they spend looking at tutorials, and whether the learners complete the 1-minute video before time is up and they move to the Exit Ticket.

Some questions for the beta test will include:

• How did the storyboard template serve to guide your understanding of creating a video?

• Was the Adobe Rush provided walkthrough useful or did you rely more on the extra tutorial?

• Are you familiar with other Adobe tools that helped you adapt to this one quickly?

  • What other video editing apps have you used that might have influenced your learning curve/been transferrable to this activity?

• What other features from Adobe Rush did you want to learn more about?

• Is asking for a 1-minute video too long/short? Were you ever overwhelmed by the learning curve for this specific activity and/or would you have wanted more time? (we must emphasize for our beta testers that it is okay if they do not finish the video)

Learning Theories

• Constructionism: Learners anchor their learning to a concrete deliverable--the 1-minute video introduction of themselves.

• Constructivism (Metacognition): Learners track their understanding of Adobe Rush with the storyboard template and reflect during the Exit Ticket. Additionally, students' prior knowledge (of either different Adobe tools and/or video editing applications) influence their learning curve to Adobe Rush and also might have provided transferrable skills they could build upon.

  • (In part 2 of the Jigsaw Activity, students utilize Social Constructivism to teach their teams the different tools they learned about)

• Culturally Relevant Pedagogy: Learners are invited to create a personalized narrative introduction with little constraint as to the specific content.

• Situativity: Learners are building community (in the workshop/with their workshop peers) based around their individualized and personalized narratives of themselves.

• Behaviorism: There is an example for students to follow on the storyboard template.

Implementation & Iteration

For this activity, our user testers were 3 young adults aged 17-20 (at the upper end of our 12-19 target demographic). Because we had to test this activity remotely, you can find the Google Slides we used for guiding our user testers below.

As stated in the evaluation plan, we originally wanted to focus our test on how learners would react to the exit ticket and if there would be a more intuitive flow to the questions (as this was a weak point in the Sign-Up Survey activity). However, our user testers were unable to reach the Exit Ticket due to time restrictions (our test could not last the full 45 minutes that we would have in the module). This in itself was a learning moment of testing prioritization, but it also showed us that we might want to reconsider the large scope of this activity.

Regardless, we gathered significant key findings into the following questions on Adobe Rush:

• Was the Adobe Rush provided walkthrough useful or did you rely more on the extra tutorial?

• Are you familiar with other Adobe tools that helped you adapt to this one quickly?

  • What other video editing apps have you used that might have influenced your learning curve/been transferrable to this activity?

• Is asking for a 1-minute video too long/short? Were you ever overwhelmed by the learning curve for this specific activity and/or would you have wanted more time?

Of our three user testers, one accidentally skipped the entire tutorial provided by Adobe Rush. Unfortunately, the app's design would not allow her to go back. The two other users were familiar with different video editing tools, so Adobe Rush's tutorial was more "like a refresher" and skimmed through. Overall, all three user testers found the app easy to use and were engaged with the task. One even said, "I'm not taking this digital storytelling class--and I've never vlogged in my life--but I kind of want to use this to make a vlog." This quote felt significant as it reminded us of a central goal to our learning experience: "to not only help students learn more about creating multimedia narratives but to encourage further, individual study after completing these modules."

Based on the above test results, some refinements we should consider include:

• Creating a protocol to allow instructors to be more intentional about steps and tips to not skip the pop-up tutorial provided by Adobe Rush.

  • Maybe this would look like explicitly stating it in directions or having all the learners go through the tutorial as a collective.

• Adding two questions to the Exit Ticket that address previous video editing experience.

  • Maybe something that asks "Did your experiences help you figure out Adobe Rush or was most of your help from the tutorial?"

• Shortening the scope of the activity to have students make a 30-second video rather than a full 1-minute.

  • This would allow the learners to focus more on the tool than on "completing" an assignment.