In 2021-22, the introduction of the digital peer evaluation tool, Buddycheck (PV2), prompted Sheffield University Management School (SUMS) to reassess its peer evaluation approach. Instead of mere tool adoption, discussions centred on the broader pedagogical significance of peer evaluation.
Recognising a potential paradigm shift, I collaborated with our Postgraduate Director of Education to lead a comprehensive review of SUMS' peer evaluation methodology (PV1). The initiative was two-fold: a review of contemporary academic literature on self and peer assessment in higher education, and an exploration of innovative peer assessment implementations by leading institutions and potential employers of our graduates.
The paramount discovery was the trend towards using peer assessment for honing individual team-working skills, employing it persistently within a continuous-improvement framework. Transitioning SUMS to this modern, sustainable formative peer assessment model would not only advance our students’ preparedness for their future careers but also impact all of SUMS’ undergraduate and postgraduate programmes.
Consequently, I formulated the "SUMS Peer Evaluation Template Proposal (using Buddycheck)" for the Education Strategic Oversight Committee (ESOC) consideration. This proposal was bifurcated: the first segment contextualised our existing stance on peer evaluation, and the second detailed the transformation into a sustainable model, drawing insights from Boud and Falchikov (2007), aligned with the Sheffield Graduate Attributes (https://www.sheffield.ac.uk/skills/sga) and SUMS’ educational objectives (PV3).
To enrich the proposal, I investigated diverse sources for assessment questions. I collated three distinct question banks, showcasing the versatility of the sustainable peer evaluation model. The first, Buddycheck's default set, emphasised the potency of simple queries. The second offered reflective insights for mapping students’ developmental trajectory (Stewart, 2014). The third embedded the assessment within a series of contextual activities (Wanner and Palmer, 2018) (PV3).
My implementation plan was structured into four phases: the creation of the SUMS Peer Evaluation System; stakeholders' buy-in; post-evaluation survey development; and pre-evaluation activity design, inspired by Wanner and Palmer (2018).
Central to the system was a question bank, drawn from my experiences in Virtual Learning Environment (VLE) training and understanding of group dynamics (Tuckman 1977) (PV1). Emphasising flexibility and relevance, questions were curated around themes like group dynamics (Boss, 2018; Fellow, 2020), individual accountability (Dagher, 2020), and teamwork abilities; all designed to facilitate maximum student participation.
The crux of the system’s potential success was its ability to gauge achievement of group work learning outcomes and secure active student engagement, even when devoid of grading stakes.
To test the system, I collaborated with three module leaders for a pilot during the Spring Semester (2021-22). This trial phase was an avenue to create and validate a comprehensive guidance manual, "Creating a Formative Peer Evaluation using Buddycheck" (PV3). Engaging 663 students in 183 groups, the pilot registered an impressive 83.8% mean engagement, with results assisting module leaders in evaluating student performance towards learning outcomes.
With ESOC's endorsement, the new peer evaluation system was instituted across SUMS in 2022-23 (PV4). To ensure a seamless transition, I conducted extensive training sessions for academics, simultaneously offering support to both students and staff. After a full academic year of implementation, I audited the system's application and presented a comprehensive report, inclusive of raw data, analysis, and forward-looking recommendations, to both Undergraduate and Postgraduate Education Committees. The Postgraduate and Undergraduate Directors of Education and I worked together to author the “SUMS Peer Evaluation Principles” policy (PV3), which was shared with the Faculty Director of Learning and Teaching, and the central Elevate team who upheld it as an example of best practice (PV4).
This novel application of Buddycheck (PV2) was showcased within the Faculty of Social Sciences, and I was invited to present it at the inaugural Elevate Core/Enhanced Symposium, spotlighting groundbreaking digital learning practices across the university (PV4). For the transformative impact on SUMS' teaching and learning landscape, the university recognised my efforts with a 'Highly Commended' accolade in the Digital Innovation category at the annual Education Awards.
A paper abstract of the development and outcomes of our new peer evaluation system has been submitted to the Assessment in Higher Education (AHE) conference, and it has been accepted by the panel for presentation (PV4).
PV2: Keeping up-to-date with New Technologies (and Technology Updates)
The university’s previous online peer assessment system, WebPA, was developed as an open source tool by Loughborough University. Lack of software updates rendered WebPA increasingly unreliable in its integration with our VLE, increasing my sense of frustration to a point where I could not responsibly recommend its use. I reflect upon this as a metaphor for the necessity of this core Professional Value, where our continually-updated VLE represents the learning technology environment, and WebPA represents a learning technologist who does not stay abreast of developments.
The official withdrawal of WebPA in 2020 due to information security concerns left SUMS without an online peer-moderated marking system, although in practice general deployment had ceased by 2018.
During that period I sought potential replacements, and identified specifically:
TEAMMATES (https://teammatesv4.appspot.com/web/front/home) by the University of Singapore. This was piloted successfully in ‘MGT134 Business Management in Context (2018-19)’ (PV2), but could not proceed further due to non-compliance with the General Data Protection Regulation (EU GDPR).
PeerWise (https://peerwise.cs.auckland.ac.nz/?nop) by the University of Auckland. Although this is also a peer-moderated marking system i.e. the quality and difficulty of the questions, as well as the feedback, are managed through peer assessment rather than by instructors alone, it could not be practically adapted for our students’ peer evaluation of group work.
We were stranded without a reliable online peer evaluation facility for group work for more than a year, until the COVID-19 pandemic abruptly forced the university to modify its assessment procedures.
When Buddycheck was officially made available for the academic year 2021-22, I was keen to assess its suitability as a replacement for WebPA (PV2). Much to my relief, it proved to be far superior on all fronts: the user interface was simple, clean and attractive; it was highly functional with just two easy-to-create question types; the instructor view of results and the feedback label algorithm were highly informative; and, as an LTI tool, it was easier to administer. Most significantly, Buddycheck possessed a database which stored previous evaluations and questions in pools, making it a tool of strategic value (PV1).
Being able to explore the software and its integration ahead of time placed me in a confident position to be able to take a third-person perspective of the Buddycheck training session, delivered by the central Elevate team to academic colleagues. There I had the cognitive capacity to assess their style of delivery and nuances of its reception around the room (PV3), and have the ‘ready mind’ for the insights which led to my review of our peer evaluation policy and practice.
PV1: Interplay between Technology and Learning
The Elevate Team’s typical acontextual feature-centred training session left my colleagues disoriented, uncertain as to how to deploy Buddycheck, and fearful of the amount of resource it would take. Their reluctance became manifest during the ‘Question and Answer’ segment; in the strong views expressed, not to the Elevate team about Buddycheck, but to SUMS’ Directors of Education about how we implemented student peer evaluation of group work.
I reflect upon this event as a scenario in psychological safety. It’s a strong example of how challenges can be framed in change management, either as: ‘implementation challenges’ which are presented as high-stakes, thus heightening the fear of failure and diminishing psychological safety; or ‘learning challenges’ which are a series of low-stakes trials as part of a continuous improvement cycle (Edmondson, A., 2019). The introduction of a new learning technology, significant update of technology, or novel application of an existing technology (PV2), demands changes in practice for adoption to occur. An ‘implementation’ mindset generates resistance by triggering a ‘fear of failure’ response, while a ‘learning’ mindset side-steps this response in the presence of strong support. It is this interplay between technological adoption and the learning mindset, which constitutes this core Professional Value, that I find most intriguing.
Knowing this, I took a contrasting approach. I reviewed SUMS’ peer evaluation methodology and, framing it as a staff development learning opportunity, used the technological adoption of Buddycheck to drive a cultural change to a formative peer evaluation model (PV1). I was relieved when, in the presence of the robust learning support provided, any initial resistance to technological adoption melted away.
Buddycheck’s database confers the ability to use previously-created evaluations as a template for successive ones; and, to tag user-designed questions within its pool, and to filter the pool as required. I immediately recognised this technical functionality for its strategic pedagogic value. By establishing and evolving a standardised pool of SUMS-specific questions for academics to select from (PV4), we would increase the consistency and comparability of our student peer evaluations intra- and inter-module. This, along with the ease of using previous evaluations as a template, made the sustainable deployment of student peer evaluations across our programmes feasible. The Buddycheck database’s facility to export data allows students’ performance to be collated and tracked across modules and years, and can provide information for SUMS’ Assurance of Learning activities.
This technology has positively impacted the sustainability of pedagogic practice in SUMS. And it was through the bidirectional learning process, between my valued colleagues and myself (PV3), that shaped its success.
PV3: Learning from a Diversity of Colleagues
Permit me to paraphrase an academic colleague who said, “when I first heard about it, it felt as if someone was putting rocks in my backpack. Now, I’m really excited.” This response tells the story of how I engaged colleagues, obtained their agreement to trial my idea, and learned from each other. Of the three academics who were available to run student peer evaluation of group work that year, all of them responded in this manner. They had been aware of the deployment of Buddycheck but, due to a paucity of information, had anticipated the worst. I tackled this by establishing the psychological safety for a collaborative learning environment, and reinforced the ability to take interpersonal risk, through a high level of support via a series of one-to-one development sessions (PV1).
Hailing from the subject areas of Marketing, Decision Sciences, and Strategic Management, I noted how each colleague learned, interpreted, questioned, and responded differently. These observations and their input proved invaluable to refining the user guidance I’d created; and to establishing the initial SUMS question pool in Buddycheck (PV3). The latter was facilitated by my counterpart in the Digital Learning Team, providing unusual insight into the software’s backroom configuration, and led to SUMS having the first bespoke-tagged question pool in the university.
Data from the trial successfully supported the proposal’s thesis. It paved the way for my collaboration with our Postgraduate Director of Education to interpret the data and strengthen the proposal ahead of her advocacy for it at ESOC (PV3). Following the adoption of my recommendations, the Postgraduate and Undergraduate Directors of Education and I worked together to produce the “SUMS Peer Evaluation Principles” document. That was a new experience for me. My leadership in innovating pedagogy through the deployment of technology has resulted in changes to, and the creation of new policies. However this was the first time I was involved in the process end-to-end, and the definition and setting of key principles, in performance and assessment, and how to frame them in a policy document, was an experience which taught me valuable lessons (PV3).
Meanwhile my learning from academic colleagues of the initial trial and the pilot continued, when the support and guidance not only continued to be refined along a similar vein, but was contextually enriched by this process. For example, I began selecting and collating (with permission) examples of communications sent to students by colleagues during the formative evaluation process, and provided them as ‘serving suggestions’ to all module leaders, thus sharing best practices (PV4).
The impact of the review on the work of my professional services colleagues was significant. By understanding how they administered student peer evaluations, working together and supporting their development (PV3), we were able to reduce the administrative burden of collecting, marking, and processing of 5326 peer evaluation scripts during the pilot year alone.
As the transformation of our student peer evaluation process gathered pace, interest spread first to departments in our faculty, and then to the wider institution, and eventually extended beyond (PV4). I continue to welcome the increased attention and its accompanying scrutiny, viewing it as a necessary means of furthering its utility and robustness.
PV4: Communicating and disseminating effective practice
The first indication of the substantial nature of the initiative, for me, came from an impromptu presentation I gave to my fellow learning technologists in the Faculty of Social Sciences. Their responses and questions validated (PV3) the premise of the completed draft proposal, and it gave me great assurance.
During the pilot, the staff development aspect of my role naturally brought me into contact with relevant academic and professional services colleagues to communicate policy, deliver training, learn from them, and disseminate best practice (PV4) within SUMS. This continued throughout the academic year, culminating in an audit of the deployment of sustainable student peer evaluation whose data was analysed and reported to our Education Committees, and then disseminated to module leaders via programme directors. Having undertaken this for a third time, the most striking aspect is the power generated by closing a feedback loop to give rise to a continuous improvement cycle.
At the same time as the central Elevate team had received our ‘SUMS Peer Evaluation Principles’ and accepted it as an example of best practice, I was invited to present my work at their inaugural ‘Digital Education Core and Enhanced Symposium’, whereupon it was deemed fundamentally significant to be categorised as ‘core’. Although the preparation for this filled me with some trepidation, feedback from attendees university-wide reassured me that the presentation was well received. Earning the ‘Highly Commended’ accolade for Digital Innovation (PV2) at the subsequent Education Awards heightened awareness of the initiative to the Vice Chancellor and Faculty Director of Education.
Following this, SUMS’ Head of Professional Services encouraged a broader dissemination of our work. With her support, I submitted a paper abstract to the ‘Assessment in Higher Education’ conference which was accepted. I look forward most to the Question and Answer session following the presentation.
MS Word document of the proposal to move SUMS to a Sustainable Formative Peer Assessment model for group work which was presented to ESOC and approved:
AA-01 SUMS Peer Assessment Template Proposal (using Buddycheck).docx
MS Word document of the step-by-step guidance for module leaders on creating a formative peer evaluation using Buddycheck in Blackboard:
AA-02 Creating a Formative Peer Evaluation using Buddycheck.docx
MS Word document of the step-by-step guidance for module leaders on batch-creating groups in Blackboard Learn Classic for use in Buddycheck:
AA-03 Blackboard Batch-Creating Groups for BuddyCheck.docx
MS Word document of the report submitted to Undergraduate and Postgraduate Education Committees upon the completion of the Peer Evaluation Audit at the end of the first year of full roll-out:
AA-04 Formative Peer Evaluation (Buddycheck) Audit Report 2022-23 Anonymised.docx
MS Word Document of 'SUMS Student Peer Evaluation Principles for Group Work 23/24' authored by the Directors of Education and Loo Yeo, which has been adopted as policy:
CA3-b3 SUMS Student Peer Evaluation Principles 2023_24 (for sharing).docx
MS PowerPoint file of the "A Move to Sustainability: Revising Peer Evaluation" presentation with speaker's notes delivered at the inaugural Elevate Core/Enhanced Symposium:
AA-05 Elevate Core/Enhanced Symposium Revising Formative Peer Evaluation Presentation.pptx
PDF of a letter from the Vice Chancellor of the University of Sheffield in congratulations on being recognised as "Highly Commended" for a 2023 Education Award in the category of Digital Innovation in Education, for revising student's peer evaluation for group work in SUMS:
AA-07 Education Awards letter51993.pdf
Screenshot of "Assessment in Higher Education (AHE) Conference 2024: Notification of Acceptance" for presentation of "Case Study: Sheffield University Management School (SUMS) Peer assessment" after having been selected by blind peer review:
AA-08 Assessment in Higher Education (AHE) Conference 2024 Notification of Acceptance.png
Screenshot of an email thread from the Digital Learning Team and Senate Fellow with feedback on the presentation delivered at the Core/Enhanced Symposium:
AA-09 Screenshot Elevate Symposium presentation thanks.png