Learning with Generative AI
Generative AI Literacy and Best Practices for Engineering Education
Embedded Files
Introduction
Introduction
Since the widespread availability of GenAI tools, increasing number of students are using these tools to complete their Homeworks and assignments. This has raised a concern on any negative impact these tools will have on student’s critical thinking and other essential learning outcomes. This project investigates the impact of Generative AI tools on undergraduate students' learning outcomes.
Research on the topic of 'Generative AI and Education' informs the following:
a. There is a consensus on the importance of Generative AI tools as they are increasingly becoming critical part of engineering outputs in the industry.
b. Multiple studies have shown the negative impact of Generative AI tools on students learning outcomes, when these tools are used excessively.
c. There is a wide range of diverse opinion on best way to adopt these tools in engineering education, from using them as companions to completely abandoning classical engineering education.
This study leveraged this analysis to design a study and an intervention to help students adopt GenAI responsibly.
Situational Factors
Situational Factors
The study was conducted with students of three courses from the 2025 Spring Semester in the college of Systems and Industrial Engineering:
A 200-level engineering course with 55 undergraduate students.
A 300-level course with 46 undergraduate students.
A 400/500 level course with 30 students, with demographics consisting of undergraduate, graduate and PhD students.
All the courses were delivered by the faculty in Hybrid mode (In-person & Online). Faculty's delivery of classes included in-class lectures, problem solving, virtual lab, designing solutions and discussing case studies.
GenAI policy, guidelines & instructions were published by the university as well as part of the course syllabus. Policy allowed students to use GenAI tools for non-graded learning but not for homework, assignments, labs or exam components. The policy focus is mainly on academic integrity and ethical use of tools.
TAR Question
TAR Question
Based on the student profiles, study objectives, research, and faculty direction, following the teaching as research question was framed as follows:
“How can we build intrinsic motivation for students of engineering courses SFWE***, SIE***, and SIE** to implement best practices while using GenAI tools in order to ensure that Essential Learning Outcomes and are not compromised?"
It was agreed with the faculty partner that it is important to try and influence the students to be self-motivated in responsible usage of GenAI tools
Approach/Method
The overall approach of the project involved following steps:
A baseline survey to ascertain current understanding of students on the topic of GenAI
A 30-minute intervention class to inform and educate on the topic of GenAI
An offline workshop with hands-on exercise
A post-intervention (lecture and workshop) survey to validate any changes in students understanding of GenAI tools
Critical analysis of survey results and workshop reflections by the students
Update Best practice guides and any changes to policy instructions based on the outcome
Survey design
Survey design
Both the baseline and post-intervention/post-workshop surveys were identical in order to enable quantitative comparison of results. The survey had 20 questions. 18 of these were based on 5-point Likert scale where students had to answer each question on a scale of 1 to 5 (Strongly Agree, Agree, Neutral, Disagree, Strongly Disagree). 2 questions were open ended and elicited long text responses. These 20 questions formed followed broad sub-groups:
Policy Awareness (To what extent are students aware of the existing policies regarding the use of GenAI tools in their learning journey?)
Responsible Usage (How knowledgeable are students about using GenAI tools without compromising essential learning outcomes?)
Industry Trend (To what extent are students aware of industry and job market expectations about GenAI related skills?)
Competency (How competent are students in using different GenAI tools and platforms?)
Use-cases /scenario based (How will students fare when tasked with use-case and scenario based questions about responsible use of GenAI?)
Intervention Lecture
Intervention lecture of approx. 30 minutes was aimed at informing the students about the GenAI technology, existing policy nuances, current academic research and whitepapers on the topic of 'GenAI and Education', AAC&U essential learning outcomes, Industry and job market trends. The content and lecture delivery was designed to help students appreciate the mandatory learning outcomes from their courses, how GenAI usage impacts these outcomes, and a demonstration of responsible use of a representative GenAI chat application like ChatGPT. Emphasis was provided during the intervention lecture to explain the AAC&U's Value Rubric and Essential Learning Outcomes. Out of the 16 Essential learning Outcomes in the AAC&U framework, three were identified as common for any engineering course (and hence could not be compromised while engaging with GenAI tools) - Inquiry & Analysis, Problem Solving and Critical Thinking.
Offline Workshop
The offline workshop was launched for all the students after the lecture intervention (informative session on GenAI literacy). A document with clear instructions was provided to students to complete a hands-on exercise. The objective of this hands-on exercise was to inform the right usage of GenAI tools like ChatGPT during the learning process of their respective course. Students completed the exercise offline and uploaded their reflections and notes on their learnings to course homework folder. This qualitative data was later analyzed along with survey responses to understand the effectiveness of this project.
This hands-on exercise had following steps:
Students logged into ChatGPT or any other GenAI tool of their choice.
Typed in a question relevant to their course. This question resembled a typical question students would get as one of their homework assignments.
Students got the response from the GenAI tool.
Students were now asked to use 'prompt engineering' technique to turn ChatGPT into a 'Tutor' mode. The suggested prompt is given below:
“Until I type ‘END TUTOR’, please act as my tutor. This means you will not give me ready-made answers and instead will prompt me to employ ‘critical thinking’, ‘Problem solving’ and ‘Inquiry and Analysis’ competencies and come up with my own answer. Objective is to make me learn and help me solve the problem myself instead of giving full answer“
Students now noted the difference in responses compared to step-3. Students also wrote down their chat experience.
Data Analysis and Results
Data that was collected in this study comprised of following:
Pre and post intervention surveys - a total of 96 responses
Offline workshop (hands-on exercise) - approx. 30,000 words reflections and notes
he survey responses were stored in a data file and analyzed with a custom python program. A predefined set of graphs and charts were produced to help visualize the results. The qualitative responses from the surveys and workshop reflections were analyzed using Python programing libraries as well as manual encoding techniques.
A comparison of baseline and post-intervention survey responses
There is a common trend for average scores across all three courses. In the baseline surveys, al the survey categories scored between 3.0 and 4.0 (between 'Neutral' to 'Agree'). In the post-intervention survey, the score improved (between 4.0 and 5.0 ie., between 'Agree' and 'Strongly Agree').
Use-case related responses have remained more or less same. This could indicate need for more sample size or more focus during intervention workshops. This means that part of the intervention lecture could be dedicated to showing use-case scenarios similar to survey questions to help students understand how to apply their understanding of 'responsible usage of GenAI' to specific scenarios.
Policy awareness, Responsible Usage and competency scores have increased indicating positive outcome from the intervention workshops (Increase up to 23%)
Only in the case of 200 level course, post-intervention survey scores have decreased in two category areas (Responsible usage and Industry Trend). This could be because of the differentials in responder numbers between baseline and post-intervention surveys. Since the surveys were anonymous, it is difficult to further analyze the anomaly. For the subsequent studies, it is recommended to design an identification mechanism of responders (without disclosing the student names or ids).
It can be generally concluded that intervention made a significant improvement in student's GenAI literacy and awareness about its responsible usage.
The rich qualitative feedback received (approximately 30,000 words) through the hands-on exercise and student reflections, point to a significant increase in the awareness, enthusiasm and motivation of the students to use GenAI tools responsibly. The study results also indicate success in making students clearly define and realize what responsible usage of GenAI means (using GenAI without compromising three essential learning outcomes of 'critical thinking', 'problem solving', and 'Inquiry & Analysis'.
Overall Summary
This 10-week project conducted with the students of three courses in the Systems and Industrial Engineering college with the aim to improve GenAI literacy and motivate students to use these tools responsibly has achieved following outcomes:
Up to 23% increase in Policy awareness, Responsible Usage and Competency in using GenAI tools for students enrolled in the courses.
A clearer understanding for students on how to be aware of and identify essential learning outcomes from their courses as well as make sure that it is not compromised while engaging with GenAI tools.
Increased enthusiasm and self-motivation for responsible usage of GenAI tools during their learning journey and data to build a best practice guide and update GenAI policies that can benefit students across the college and university.
Recommendations
Based on the experience from this project, following recommendations are made:
Build and provide learning materials about GenAI literacy to students enrolled in every course. Ensure that learning materials include industry trends, guidance on essential learning outcomes from the enrolled courses we well as practical guides on responsible usage of GenAI tools.
Update the AAC&U framework with additional ELO (Essential Learning Outcome) and Value Rubric for GenAI Literacy.
Collaborate, share, and synchronize best practices with similar initiatives on AI and GenAI literacy across colleges and universities.
Acknowledgements
I thank Dr. Lisa Elfring and Dr. Kristin Winet for guidance and support throughout this project. Without their mentorship, this research project would not be successful. I am grateful to Prof. Sherilyn Keaton for the providing opportunity, access, her valuable time and guidance to design and successfully execute this project.
About the Author
About the Author
Aditya Bandimatt is a graduate from School of Information Science, University of Arizona with specialization in Machine Learning (Spring 2025). He is interested in interdisciplinary applications of AI to Systems Engineering and Social Science domains. He is also a certified Product Manager and DevOps Architect with extensive consulting experience in digital transformation, AI adoption and AI product management.
References
Vassilka D. Kirova, Cyril S. Ku, Joseph R. Laracy, and Thomas J. Marlowe. 2024. Software Engineering Education Must Adapt and Evolve for an LLM Environment. In Proceedings of the 55th ACM Technical Symposium on Computer Science Education V. 1 (SIGCSE 2024). Association for Computing Machinery, New York, NY, USA, 666–672. https://doi.org/10.1145/3626252.3630927
Ozkaya, I., Carleton, A., Robert, J., and Schmidt, D., 2023: Application of Large Language Models (LLMs) in Software Engineering: Overblown Hype or Disruptive Change?. Carnegie Mellon University, Software Engineering Institute's Insights (blog), Accessed January 28, 2025, https://doi.org/10.58012/6n1p-pw64.
Schmidt, E. (2023, June 1). Eric Schmidt full controversial interview on AI revolution (Former Google CEO) [Video]. YouTube. Retrieved from https://www.youtube.com/watch?v=mKVFNg3DEng (34:00 min to 36:00 min)
Matt Welsh, The End of Programming, ex Professor Computer Science, Harvard University: Welsh, M. (2023, January 1). The end of programming. Communications of the ACM. Retrieved from https://cacm.acm.org/opinion/the-end-of-programming/
Microsoft says AI tools are affecting critical thinking at work: Kobie, N. (2025, February 18). Microsoft says AI tools such as Copilot or ChatGPT are affecting critical thinking at work. ITPro. Retrieved from https://www.itpro.com/technology/artificial-intelligence/ai-tools-critical-thinking-reliance
Bold, U. (2025). Mark Zuckerberg predicts AI will replace software developers. LinkedIn Pulse. Retrieved from https://www.linkedin.com/pulse/mark-zuckerberg-predicts-ai-replace-software-developers-bold-unxac
Vatican Press Office. (2025, January 28). Vatican on AI and education. Retrieved from https://press.vatican.va/content/salastampa/it/bollettino/pubb
McKinsey & Company. (2025). How an AI-enabled software product development life cycle will fuel innovation. Retrieved from https://www.mckinsey.com/industries/technology-media-and-telecommunications/our-insights/how-an-ai-enabled-software-product-development-life-cycle-will-fuel-innovation
Thilina. (2025). Software engineering in the era of generative AI: Do we still need to code? Medium. Retrieved from https://medium.com/@thilina.hci/software-engineering-in-the-era-of-generative-ai-do-we-still-need-to-code-82307f6f9fda
Schwartz, E. (2025, March 14). Coding AI tells developer to write it himself. TechRadar. Retrieved from https://www.techradar.com/computing/artificial-intelligence/coding-ai-tells-developer-to-write-it-himself
Effects of GenAI usage on university students: Abbas, M., Jam, F. A., & Khan, T. I. (2024). Is it harmful or helpful? Examining the causes and consequences of generative AI usage among university students. International Journal of Educational Technology in Higher Education, 21(10). https://doi.org/10.1186/s41239-024-00444-7
Bastani, H. (2025). Generative AI can harm learning. Retrieved from https://hamsabastani.github.io/education_llm.pdf
American Association of Colleges and Universities, AAC&U Essential Learning Outcomes https://www.aacu.org/trending-topics/essential-learning-outcomes
Page updated
Report abuse