The CMALT Guidelines state that:

"Statements should show how you have found out about learners’ needs and the context for their studies, and how you have developed approaches that reflect this. [...] Evidence of changed practice, rather than simply the recognition that this is an important area, is required."

A major aspect of my current role is to develop a suite of new online self-study academic skills resources. This is a large-scale, ongoing project. As part of the planning stages of this project, I've created a set of online resources for the 2019/20 MSc Psychology in Education (MSc PiE) cohort. I also provide weekly face-to-face support for this cohort, but the resources were presented as self-study resources, rather than as blended learning activities. See the Specialist Option for details of the wider project and discussion of the resources themselves.

In this section, I discuss how I've learnt about students' needs and preferences, how I applied this to develop resources for Term 1, and how I'll take this forward to inform resource development for Terms 2 & 3 and the ongoing online self-study resources project.

Who are the students?

There are 54 students in the cohort. Most are returning to study after a period outside education - I'd estimate the average age as early-mid 30s. The tutor for last year's cohort (which had a very similar profile) told me that the time away from studying meant that many students weren't very confident in their academic and writing skills.

The majority are home students. There are 15 from a non-English background (China, SE Asia, Egypt & Europe) and 4 from a non-UK English background (USA & India). This means students have a range of previous educational experiences, and so may have differing academic support needs.

Although MSc PiE is science-focused, it's a conversion programme, so it's assumed students have no previous psychology or science background. This is largely the case; students mostly come from working in an educational context (nursery, primary school, secondary school), with some from a social sciences or humanities background. This means that almost all are new to studying science and the genre of scientific writing.

What do students need to do on the course?

I used a few methods to learn about module requirements for Term 1:

• used module VLE sites to find module content and assessment tasks.

• used the course handbook to find assessment deadlines, marking criteria and general guidance from the department.

• attended initial lectures of the core modules.

• clarified assessment task requirements with module tutors where necessary.

Using this to inform online self-study resource design for Term 1

Based on the needs analysis, my priorities were that online resources help students develop the skills needed for scientific study and writing, build their confidence and cater to varying needs.

Reacting to module requirements & initial needs analysis

I used the concerns and suggestions from the needs analysis and details of module assessments to identify general skills topics and assessment-specific topics for the online resources. For example, a major need is to develop skills and confidence in science writing, so I created an interactive tutorial to raise awareness of the features of science writing and help students build their confidence through interactive tasks with automatic feedback. I also created numerous resources that specifically relate to completing formative and summative assessment tasks.

I used analytics to get an idea of how students used the self-study resources on the academic skills VLE site during Term 1 and the holiday period.

Panopto has good analytics, showing each user’s viewing activity, and the VLE shows the number of clicks per content area and time spent on the site. The UoY doesn’t currently have the ability to track use of Xerte tutorials or Google documents directly, so I provided bitly links to these to track the number of clicks. However, all resources are embedded on the VLE site, so direct usage on the site can’t be tracked.

18 Resources were clicked or viewed 345 times in total (one text document was untracked), with use ranging from 1 to 65 clicks/views. The average time spent on the VLE site was 3.89 hours, with a maximum of 11.41 hours. Usage continued through the term break, showing that students using the resources as reference materials while preparing their assignments.

To gather feedback on the online resources, I sent out a survey at the start of Term 2. This time frame gave students the chance to use the self-study resources while preparing Term 1 assignments. There were 23 responses, with an even split between home and international students (42.5% of the cohort). 91.3% of respondents reported that they'd used the online resources.

Location of resources

Most respondents would like resources to be located in a specific academic skills VLE site (78.3%), with integration into degree module sites also fairly popular (47.8%). Few wanted to access resources through the Skills Guides (17.4%).

Improving resources

There were 6 suggestions on how resources could be improved. There were a couple of requests to make a tutorial for each module, and both respondents who didn't use the resources also suggested tailoring resources to modules more specifically. There were resources that were directly relevant to the assessments in each module, but these were labelled by assignment type, rather than module (multiple modules had the same broad assignment type). It seems that more explicit signposting to relevant modules may be beneficial.

Other suggestions were to provide more practice activities and downloadable text-based versions of tutorials, and also to put resources in one central location (this refers to the emailed links before setting up the VLE site).

Evidence:

• 2b_MSc PiE online resources feedback survey (note: the survey is still live, so there may be additional responses not included in the results above)

I also conducted a focus group with four students early in Term 2 to gather more in-depth insight into perceptions of online resources. All participants were home students from a non-science background, and had been out of education for a while before starting the course.

Here's a summary of the key points from the discussion with some representative comments.

"I've never done APA referencing before. And the way you broke it down in the slides and gave examples, I needed that to double check my referencing. I found I kept going back to those slides to make sure I got it right."

"I think when we have a lot of other stuff to read and do, to have somewhere you know you're going to get a really straightforward, simple, concise answer is like a bit of a breath of fresh air."

"[...] to have more examples of what a good piece of writing looks like would be very, very helpful for me."

"I think [...] in general, I really like how concise the information is. There generally just tends to be just eight to 10 slides, and I think that that's a really manageable amount. And then what's on the slides is very useful. There's not a lot, there's nothing that I feel isn't useful."

"Yeah, I tried [the interactive tasks], and it made you feel like you were on the right lines before having to find out what the correct answer was. And I used it as a sort of like a reference guide. So it wasn't, sort of like information but it was like a reference point to store and go back to again and again."

"Or even if [interactive tasks are] optional. There could be a hyperlink, and say if you want further practice, and then it's up to you then to decide. "

"I generally prefer to have something to read rather than a video."

"I'm not suggesting that you don't put them in, because other people might prefer to watch somebody talk and make notes from somebody talking. "

"Like where you are showing somebody how to do something? [...] If someone showed me on a video, I'd watch that. But I don't want a lecture, I haven't got an hour. I've got five minutes to find out how to do this."

"It's definitely good having it in one place on the VLE rather than in each module or anything."

"I think there was a point where [I'd] seen something you'd done, but I couldn't remember where it was [...]. So in that case it would have been easier for me just to go to the [core module] site and find it in that."

Evidence:

• 2b_MSc PiE online resources focus group (transcript)

Here I'll consider the implications of evaluation outcomes for developing online self-study resources for MSc PiE during Terms 2 and 3 (for reference, as I write it's early in Term 2). For discussion of how this will also inform the wider online resources project, see the Specialist Option.

Feedback suggests that the general approach taken in Term 1's online resources has been effective; feedback has been very positive, and resources have generally been well-used. It seems that there aren't any major changes needed, but there are some tweaks that can be made.

Assessment-specific and general skills resources

Usage and feedback showed a strong preference for assessment-specific resources. Based on this, the priority is to develop resources relating to the essay and poster assessments in Term 2, and also preparing for the dissertation. Feedback suggested that signalling which modules these relate to will be useful, and maybe even creating separate resources for the core and optional module essays, if content is distinct enough. It could also be useful to add a VLE page that lists assignments and relevant resources to offer some guidance on resource choice.

The focus group also suggests that resources are mostly used 'just-in-time' when the specific need arises, rather than to develop skills before starting assignments. This shows that I should signpost more clearly how resources that develop scaffolding skills (e.g., paragraph structure, using source information effectively) are relevant to the remaining assignments. This could be done by adding a page at the start of Xerte tutorials, or short descriptions where resources are embedded on the VLE.

Resource format

Feedback suggests that highly relevant content is more important to students than surface features like interactivity. Although having a variety of resource types wasn't ranked as being particularly important, feedback shows that there is a range of preferences. An aim of the self-study resources is that students are motivated to use them, so continuing to provide a mix of tutorials, screencasts and text-based resources seems appropriate to cater to all preferences. Xerte tutorials were used most heavily, so it seems that using these as a starting point is a good approach, even though screencasts were rated more favourably in the survey.

Although text-based resources weren't used heavily, there were requests for more downloadable content. I have created some text-based alternatives to tutorials and screencasts (e.g., Slides to accompany a screencast, a Doc summary of the science writing tutorial), but not for every resource. Going forward, where it's feasible to create a text-based downloadable alternative, I'll try to provide one.

In relation to screencasts, usage data and the focus group findings suggest that videos should be very short, with key information fronted. This will help students access what they need more effectively. Weeks and Putnam Davis (2017) suggest limiting videos to 1-2 minutes, but I'm not sure how feasible this is. Screencasts may be more effective for demonstrations and complex explanations rather than other topics, so this should influence my choice of resource formats.

Follow up with the Department of Education

The focus group requested that online examples of good student work are available online in addition to in the face-to-face sessions. During Term 1 I asked to be able to use short extracts of example assignments in tutorials as student found them so useful, so there is some limited scope to do this. I'll try to incorporate as many short examples as I can in resources created for Terms 2 and 3. Unfortunately, I'm not currently allowed to provide complete example assignments online, so I'll put the case forward to the Department again.

I'll also approach module leaders to discuss whether it's feasible to embed assessment-specific resources on module VLE sites as well as the academic skills site to give students another way to access the resources.

Evidence

• 2b_How to… Write in scientific style (text-based alternative to interactive tutorial)

• SO_Advice on... Critiquing a research paper_(Xerte) (short examples of student work on slides 7 - 10)

Need analysis isn't over once the course starts, and it's important to react to students' evolving needs. For Term 2, I've used many of the needs analysis methods mentioned above to plan session content; using module VLE sites, assessment brief and also asking students what they'd like to cover. I haven't planned the online resources for this term yet, but content will be drawn from this. For example, the optional modules have an assessed poster presentation, which is a new task for the majority of students. They have a workshop on the technical aspects of poster creation (by a member of the Library team), so I'll create a tutorial on developing the content for a poster. During Term 1, I found students' questions to be really valuable to assess their needs and inform resource development, so I'll continue to use this method in Terms 2 and 3. I'm really glad that these students are feel comfortable to ask questions and make suggestions.

In Term 1, due to time constraints online resources were often created after a session to provide a reference guide or expand content. I'm hoping that in Terms 2 and 3 I'll be able to flip this around, and create resources before sessions. I'm hoping that this will increase the value of resources, and mean that sessions can be reserved for more collaborative, experiential tasks to practice applying the skills.

Evidence

• 2b_MSc PiE course outline (showing session content for Term 2 based on another round of needs analysis)