Institution: Queen’s University Belfast, Northern Ireland
This project took place with students on the MA in Arts Management, Queen’s University Belfast as part of a Research Methods course including critical thinking skills.
Students generated a response to a provided prompt using ChatGPT 3.5. They varied the prompt to generate two additional responses and completed a critical evaluation of the GenAI responses. The exercise was designed to enable students to better understand how GenAI functions as a tool and its strengths and limitations. By including student-generated responses and development of prompts, the format helped develop critical thinking skills and emphasised students’ own level of knowledge of a topic compared to a GenAI tool. The subject-specific (and interesting!) prompts within the students’ knowledge were chosen to allow critical engagement with GenAI responses.
Our usual pedagogical approach to teaching at the level of taught masters is driven by emphasising independent thinking, warnings about plagiarism and scaring students about Turnitin percentage scores. As lecturers, we encourage the use of Google and other search engines to get resources for discussion. Digital writing tools have been around for quite some time, but now there is merit in thinking of GenAI as a helpful, albeit foolish tool, one to be treated with both care and suspicion. In this way, this GenAI exercise was innovative in that it actively encouraged the use of a tool that might be considered a form of plagiarism. The exercise was driven by a desire to demystify ChatGPT and ensure students approach it as another tool, to be picked up, put down, and used to help the student develop something of their own.
The exercise was accessible to students as the prompts and evaluation framework were provided in advance to enable students to generate and critically analyse the responses in their own time. The task is scalable and adaptable to different cohorts of students including undergraduate, as the prompts can and should be made subject specific. The student exercise is both in the generation of new responses and in the critical analysis of the responses. The exercise was resource heavy in use of staff time, because of the need to secure ethics approval for publication. Given the innovative nature of GenAI in this context, ethics approval took some time.
This project was innovative as a learning exercise. The project was clear and accessible to students and made them think critically about ChatGPT’s responses and how to revise these prompts. It provided students with a guided entrance into using GenAI as a learning tool for research and a way to evaluate the results. Student feedback demonstrated that for those who had no prior experience with GenAI and had avoided it due to plagiarism concerns, this project provided a clear introduction to its use.
The provided prompts covered the following topics:
Is hip-hop a valuable art form?
A blog post on creative industries work in Northern Ireland
Critically interrogate Eurovision Song Contest as a cultural and political phenomenon
Key debates on colonialism and museum exhibition practices
Gender & labour issues in K-Pop.
The participating students developed really insightful additional prompts, for example a student using the hip-hop topic asked the same question but from the perspective of someone with absolutely no interest in music.
This project made the limitations of GenAI clear and showed that its responses should be taken with a grain of salt, as the responses did not include any critical evaluation of the information. Students gained experience using GenAI and possible applications for this tool. Staff saw how GenAI as a pedagogical tool is not necessarily negative and can be built into critical lecture spaces. The institutional value can be seen in how we can critically interrogate AI generated knowledge.
Seven students completed a short evaluation of generated responses. Critical insights are seen in the comments from students:
‘Only suitable for understanding common sense knowledge’ + ‘only to help researcher’s inspiration’
‘Text feels robotic’
‘Very vague’
Criticisms on lack of ‘following instructions’:
‘Didn’t include academic resources as requested’
‘Didn’t look like a blog post’
Lack of recent information:
‘Doesn’t mention as much in recent years’
‘Sources are from late 90s/early 2000s’
‘All sources were over a decade old’
Some positive responses:
‘Definitely useful information’
‘Common-sense knowledge was provided’
‘Wording and information improved in later responses’
The students found the exercise interesting in giving them the opportunity to critically engage with ChatGPT. Queen’s University Digital Scholarship hub provided useful feedback on the ethics process. Overall, this is a project that will definitely be repeated.
Maria O’Brien
Maria O’Brien is Lecturer in Taxation between the J.E. Cairnes School of Business and Economics and the School of Law in University of Galway, Ireland. At the time of research, Maria was a member of the Digital Scholarship Hub and School of Arts, English & Languages ethics committee, Queen’s University, Belfast.
Ethan Lay
Ethan Lay is a Queen’s University Masters Student studying Arts Management and Cultural Policy. He is interested in the discourse surrounding GenAI and its impact on arts and culture.