Core Area 1: Operational Issues

1a) An understanding of the constraints and benefits of different technologies

Description

Working as a Learning Technologist means I come in to contact with a great variety of technologies, all of which fit different purposes. One of the interesting challenges is to figure out which technology to choose. There is an increasing interest in integrating group work and peer feedback into assessment for a number of reasons, including the improvement of student communication skills, active engagement with material, but also to reduce workload for academics. This past year, I have worked directly with an academic to experiment with and introduce a complex peer review set-up into their course, which allows students to use feedback from their peers to review their essay plans and alter them for the final assessment.

What I first did was gather the requirements, meeting with the academic to get a full understanding of what was needed. The requirements were as follows:

·       Creation of an assessment dropbox for an essay plan

·       Students provide two sets of feedback to their classmates

·       The feedback is entirely qualitative and does not require numerical scoring

·       The peer review is anonymous

·       The academics can then review the provided student feedback and provide a numerical score to the quality of the feedback provided

There were two main contenders in terms of technology chosen: Blackboard Learn Assignments and Turnitin Assignments. As the submission itself was a fairly standard file submission, there was no need to use a tool outside of the Virtual Learning Environment for this. I experimented with using Peer Mark in Turnitin as it was the academic's initial preference, however I discovered that the set up was clunky and the limitations of the LTI integration with the VLE meant it could be confusing for students. It is my priority that the learning environment is as simple and intuitive as possible for students, so each decision I make takes this into account. However, it is School policy to use Turnitin wherever possible, so initially I honoured this and designed a workflow using the system as it incorporated similarity checking.

Evidence

After proposing this workflow using Turnitin, I discussed the requirements further with the academic and he informed me that anonymous marking and similarity checking were no longer essential, and students would not need to make use of anything other than a feedback summary, therefore I suggested that Learn Assignments would be the better option. I was glad of this, as from a support perspective, the set up is easier and more sustainable, and the students would likely find it less confusing due to the improved UI.

Reflection

The main benefit of implementing Learn Assignments was that student and staff experience using the tool has definitely been prioritised. It was easy for the academic to understand what could be done and they are able to take responsibility for setting up parts of the area, as it is not overly complex. From a student perspective, the user interface is simple and clear and the modern aesthetics mirror those they may be used to on other systems. This is unlike Turnitin, where the peer mark area is in a less intuitive area and could potentially discourage students from undertaking the task.

A constraint of Learn Assignments however is that certain setting combinations cannot be used in conjunction. For example, anonymous assessments and peer review cannot be combined, which for some assessments could prove problematic. To mitigate issues with this, I clarified with the academic exactly what was required and what the system could achieve, setting expectations as early as possible, and then we decided the way forward together to avoid surprises. If I were to undertake this process again, I would clarify with the academic at the beginning and gain a full understanding of the requirements – for example whether similarity checking or anonymous marking is required – so that time is not wasted investigating an irrelevant path. It has made me think about what we can do when the systems do not meet teaching requirements. As we are limited with what systems we have available, sometimes compromises do have to be made. However, I would consider searching for tools through the University that have completed a data protection impact assessment. In cases where many academics wish to undertake a certain type of teaching, I would look to the market and make a case for a new software to be procured.

1b) Technical knowledge and ability in the use of learning technology

Continuing with the above example of qualitative feedback in Learn, in order to ensure I was suggesting and supporting the best option, I undertook various tests in my Playground area on Learn and ran through these with the academic. The finalised set-up I proposed was:

Two Learn Assignment submission areas, one with peer assessment and one without.

The suggested workflow was as follows:

·            Students submit to the assessment

·            After the due date, they re-enter the same assessment to complete the peer review

·            Students complete the peer review by attaching a file and submitting; they then exit the assessment and submit this file to the second Learn assignment area.

Evidence

The above screengrab shows the two areas to which the student will submit. The top dropbox is for the essay plan and the peer review, the second drop box is so the tutors can review who has undertaken the peer review and provide comments and grades for the participation mark.

Reflection

Once the first run through was finished, the academic and I discussed what happens when students do not undertake the peer assess element. Although they can be encouraged, there is no way to ensure all students receive peer reviews and therefore there are some students who are missing out. I mitigated this by suggesting that tutors taking the role of the students in these circumstances and providing feedback to those who had not yet received any. It is not a perfect solution and does require tutors to engage in a way not previously anticipated, however the workload still is far less than if the tutor had to provide feedback on every essay plan. It also means that students who do not participate are flagged at an earlier point and steps can be taken to encourage engagement from their part and issues can be resolved wherever possible. This was something I wish I had anticipated before, and in future iterations, it will be considered earlier on in the process.

Something which came up after the entire process had taken place was that grades would need to be posted in order for student feedback to show up. The tutor did not want there to be a numerical grade associated with the essay plan itself, only for the feedback given, as a form of participation grade. I dealt with this issue swiftly by suggesting and implementing a Completed/Not Completed Schema in lieu of a points schema to Learn, so students could see their feedback without a numerical score being assigned. It meant we could quickly allow all the feedback to be posted without student confusion over getting a mark for a formative essay plan. What this issue has shown me is how undertaking a full test from start to finish can benefit the process as speed bumps like this can be dealt with early on. However, it also proved to me that when a virtual learning environment is designed well, workarounds can usually be found. I believe that Learn has many areas that can be improved, but in this case, I made use of the available tools to create a scenario which benefited both students and academic staff.

1c) Supporting the deployment of learning technologies

Description

I ensured throughout this entire process that I was available for contact during the deployment of the assessment and I maintained regular catch-ups with the academic to ensure the process was running smoothly. I learned a lot working with this academic, as it was a great insight into exactly how technology can improve student experience and scaffold their learning. A quote from the academic states:

“The students seemed to find the peer review element very rewarding and I would use this again in future. The process of identifying strengths and weaknesses of peers’ essay plans and writing constructive feedback based on the criteria I gave them really helped them to improve their final essays.”

What this quote evidences is that the peer review element encouraged students to actively participate in their course and that they were able to benefit from each other’s feedback in a way which improved their final grades. It also shows how the implementation and support of learning technologies can supplement tutor feedback and improve student experience. Adding a numerical grade to the quality of the student’s feedback also encouraged them to fully participate, as they knew it would affect their overall grade.

Evidence

Quote from academic

“The students seemed to find the peer review element very rewarding and I would use this again in future. The process of identifying strengths and weaknesses of peers’ essay plans and writing constructive feedback based on the criteria I gave them really helped them to improve their final essays.” 

Reflection

In the future, I would follow an almost identical procedure, maintaining the two Learn assignments and avoiding Turnitin. As Learn is consistently being updated, it may well be that the procedure would change slightly to accommodate new features so a full test would always be required before repetition. If anonymity was required, I would figure out with the academic why this was a requirement and, if a non-negotiable, I would look to find an alternative using the systems we have available. I would work with and support the academics to refine and improve the peer assessment guidance, so that no students are missing out on the experience due to lack of understanding. To avoid the issue of students not participating and thus their colleagues missing out on feedback, I would communicate to all tutors on the course the importance of checking students who have not received two reviews, and then following up with the inactive students and the reviewee to ensure they do receive useful feedback and are not negatively affected. In summary, I would prioritise clear and concise communication with all parties so that expectations are met from both a student and staff perspective. I am also interested to see how the systems we support will change over the coming years and, as peer review is becoming ever more popular, I hope that the tools available will reflect this shift in assessment and feedback.

I feel that this experience has prepared me to solve operational issues and work with strict criteria provided by academics. Collaborating with the academic has also been greatly rewarding as it has given me an understanding of how Learning Technologies can improve staff and student experience. Through solving these issues and being flexible in approach, this experience has improved my resilience and creativity as a Learning Technologist.