Consent to Act as a Research Subject

Towards an LLM-Based SmartLearningHub for Higher Education

Study Overview

This research study is being conducted to explore the design of an AI-powered SmartLearningHub in educational settings in order to assess its impact in the classroom and improve future iterations.

We are inviting you to participate in the study because you are a UCSD student currently enrolled in CSE 8A with Professor Soosai Raj and Professor Lerner. This document explains the research so that you may make an informed decision about participating.     

The purpose of this study is to evaluate an existing design of the SmartLearningHub. This study will investigate using and perceiving the SmartLearningHub in various educational contexts, including instruction, completing assignments, asking for learning assistance, interacting with the rest of the class, and creating educational materials.

The study will take up to a quarter integrated into the curriculum of the course that you will take. Some tasks are already assignments you will complete for the course, including but not limited to: periodic reflection surveys and home assignment questions such as multiple-choice questions, short responses, essays, and computer programs. Other tasks are optional and separate from the course assignments, such as an interview. Details of the tasks are listed in Section “What happens if I take part in the research?” below. Your participation involves only agreeing to let us use your anonymized data in our analysis. 

The most common risks or discomforts of this study are: (1) the loss of confidentiality, (2) fatigue or frustration during the study, and (3) the potential for AI-generated inaccurate information and other non-deterministic behavior in the SmartLearningHub. The risks induced by the study should be no more than the similar risks you will encounter in daily lives.

A complete listing of possible risks and discomforts associated with this study can be found in Section "What are the risks and possible discomforts?" below; we do not foresee any risk more serious than the risks listed.

We cannot promise any benefit to you or to others from you participating in this research.  However, possible benefits include an opportunity to get help from state-of-the-art AI-based systems designed specifically for education, reflect on your experience with AI-based tools for education, and solicit hidden insights. The investigators may learn more about how to design effective AI-based tools for the classroom, and society may benefit from this knowledge.

The alternative to being in this study is not to participate. Choosing not to participate or withdrawing will result in no penalty or loss of benefits to which you are entitled, and will have no impact on your grade in the course.

More detailed information about this research study is provided below.

How many people will take part?

There will be approximately 1000 participants in this study. The research will include students currently enrolled in CSE 8A taught by Professor Soosai Raj and Professor Lerner.

What happens if I take part in the research?

One or more of the following tasks will happen to you if you agree to be in this study:

1. You will use the SmartLearningHub for course-related activities. We will collect some usage data about how you use the tool in the background, including: how you interact with the AI agent, performance of the agent, correctness of your solution if you use an AI agent for homework help, how you rate the quality of each interaction, etc.

2. We will ask you to fill out a few surveys with questions about your background and your experience with the SmartLearningHub. The survey will be handed out along with the periodic reflection surveys for the course. It will take about 5 minutes to complete a survey.

3. We may invite you for an optional interview to reflect on your experience with the SmartLearningHub in more depth. If you choose to participate, during the interview, we will record audio of the interview. You can choose not to take the interview while still being in the study.

As you read this form, ask questions if something is not clear.

What are the risks and possible discomforts?

Participation in this study may involve risks or discomforts as follows:

Risks of Fatigue or Frustration during the Study: To minimize this risk, you will be able to (a) end the study at any point (b) ask questions for clarification to reduce the level of frustration (c) take breaks as needed. The risks of fatigue or frustration induced by the study should be no more than the similar risks you will encounter in enrolling in a course.

Risks of Loss of Confidential Information: We will collect: (a) electronic logs of your actions while using the SmartLearningHub; (b) online discussions related to the SmartLearningHub within and outside of the platform (c) survey responses; and (d) audio/video of an interview session. There is also a risk that information about you could be released to an unauthorized party. To minimize this risk, we will store all digital data on secure servers to which only the researchers in the study will have access. Data will be anonymized to the extent possible. Paper surveys will be stored in locked locations. For the audio/video recordings, you will be allowed to stop the recording at any time and you will be allowed to erase portions and/or the entire recording upon request. We will use the video and audio recordings to understand under what conditions the SmartLearningHub helps with education, under what conditions it does not help, and why. The risks of loss of confidential information induced by the study should be no more than the similar risks you will encounter in daily lives.

Risks of the AI-Generated Inaccurate Information: Because the SmartLearningHub is powered by AI, AI agents in the system may generate inaccurate information. To minimize the risks, we will validate AI-generated responses using automated approaches when applicable before releasing them to you. The instructional staff and the development team will monitor the operation of the system to ensure it is providing correct responses. You will be able to flag any suspicious AI-generated content for the instructional staff’s review. The risks of encountering AI-generated inaccurate information in this study should be no more than the similar risks you will encounter when using products such as ChatGPT.

Risks of Unknown Risks and Incidental Findings: Because this is a research study, there may also be some unknown risks that are currently unforeseeable. You will be informed of any significant new findings.

How will information about me be protected?

While we cannot guarantee complete confidentiality, we will limit access to information about you.  Only people who have a need to review your information, documents, or specimens will have access.  These people might include:

• Members of the research team and other staff or representatives of UCSD whose work is related to the research or to protecting your rights and safety.

• Representatives of Federal and other regulatory agencies who make sure the study is done properly and that your rights and safety are protected. 

To ensure the security of storing the personal identifying information and maintain confidentiality of each subject’s identity, data will be stored in secure systems to which only the researchers will have access. All other data collected during the study will be anonymized to the extent possible. You will be given IDs, which will be used to identify your data in our records instead of using your personal identifiable information. Any published results will also anonymize your information. Regarding any video or audio recordings (only applicable to interview tasks), you may ask to stop the recording at any time and to erase portions and/or the entire recording. We will use the video and audio recordings to understand under what conditions the software development tool you use is effective, and under what conditions the tool is not effective, and why. We will keep the data indefinitely to enable further research opportunities.

The results of this study may be published once the study is completed. However, we will keep your name and other identifying information confidential. We expect this study will be completed in 4 years. This is only an estimate and the actual time to complete the study may be longer or shorter depending on a number of factors.

What if I agree to participate, but change my mind later?

Participation in research is entirely voluntary. You may refuse to participate or withdraw or refuse to answer specific questions in an interview or on a questionnaire at any time without penalty or loss of benefits to which you are entitled. If you decide that you no longer wish to continue in this study, you will be required to let the researchers know as soon as possible by filling out the form linked at the end. If you are currently enrolled, or will in the future enroll, in a class taught by any of the investigators, your decision to participate or not participate will have no effect on your activities in the class, or your grade in the class.

In addition, the PIs may stop the study or remove you from the study at any time, even if you would like to continue. This could happen because the PIs feel it is in your best interest or the best interest of the study, or you do not follow the instructions given you by the study personnel.

You will be told if any important new information is found during the course of this study that may affect your wanting to continue.

What will happen to information collected from me?

The data we collect with your identifiable information (for example, your email address, voice, or face) as a part of this study may be used to answer other research questions or may be shared with other investigators for other research. If we do so, we will remove all identifiable information before use or sharing. Once identifiers have been removed, we will not ask for your consent for the use or sharing of your data in other research.

What are my responsibilities if I take part in this research?

If you choose to participate in this study, it will require no additional responsibilities to your usual activities in CSE 8A. 

Your Consent

If you consent to participate in this study, no action is needed. If you DO NOT consent to participate in this study, or you choose to opt-out at any time during the quarter, please sign and submit this form.

IRB Study #811174