Theme and goals:

The notion of intelligent systems that “care” about students is at the center of ITS research [1-3]. A variety of adaptive learning systems that “care” have been developed in the past [4, 5]. These systems make use of student/user models to adapt their interactions to a particular student (e.g., amount and type of feedback, content sequencing, scaffolding, and access to visualization tools and other materials). Student model variables include cognitive abilities, metacognitive skills, affective states, and other variables such as personality traits, learner styles, social skills, and perceptual skills [5].

Caring assessment systems are defined as systems that provide students with a positive assessment experience while improving the quality of evidence collected about the student’s knowledge, skills and abilities (KSAs) [6]. Taking a test is typically a stressful situation, and many people underperform due the stress. Caring assessment systems take into account assessment information from both traditional and non-traditional sources (e.g., student emotions, prior knowledge, and opportunities to learn) to create situations that students find engaging, and to collect valid and reliable evidence of students’ KSAs.

Taking a test is not just a passive mechanism for assessing how much people know. It actually helps people learn, and it works better than a number of other studying techniques [7]. Caring formative assessment can be done by a computer system or by peer-learners. Learners testing each other in a friendly, collegial and constructive way, can be an engaging and effective form of collaborative learning and preparation for assessment that also helps establish peer-mentorship relationships among learners. Developing systems or approaches (e.g. games) that support learners test each other in a friendly, collegial and constructive way, is a new and promising direction of research.

This workshop provides a great opportunity for ITS and assessment researchers to share information about the potential of applying ITS techniques and approaches in the development of a new generation of caring assessments. Examples of ITS technologies that have been successfully used for assessment purposes include automatic scoring of essays and short responses [8]. The use of dialogue systems for assessment is being explored [9, 10].

This workshop is a timely and relevant event for the ITS and assessment communities. New assessments for skills such as problem-solving, collaboration, and scientific inquiry include the use of highly interactive simulations and collaboration with artificial agents. Advances in ITSs will play an important role in the development of the next generation of assessment systems.

Relevant Questions/Topics:

• What ITS technologies and approaches are available for the creation of caring assessments?

• How should the assessment system handle different affective states?

• How should the assessment system adapt the interaction based on students’ characteristics?

• Alternative forms of testing for formative assessment.

Accepted Papers (Proceedings):

• When Should an Adaptive Assessment Care? (Blair Lehman, Jesse R. Sparks, and Diego Zapata-Rivera)

• Incorporating Emotional Intelligence into Assessment Systems (Han-Hui Por and Aoife Cahill)

• Diagnostic Assessment of Adults’ Reading Deficiencies in an Intelligent Tutoring System (Genghu Shi, Anne M. Lippert, Andrew J. Hampton, Su Chen, Ying Fang, and Arthur C. Graesser);

• Tower of Questions: Gamified Testing to Engage Students in Peer Evaluation (Nafisul Islam Kiron and Julita Vassileva)

• Exploring Gritty Students’ Behavior in an Intelligent Tutoring System (Erik Erickson, Ivon Arroyo, Beverly Woolf)

• Disengagement Detection Within an Intelligent Tutoring System (Su Chen, Anne Lippert, Genghu Shi,Ying Fang, and Arthur C. Graesser)

• Assessments That Care About Student Learning (Stephen E. Fancsali and Steven Ritter)

Workshop schedule (June 11th, 2018):

• 1:30pm - 3:10pm: Session 1 (4 presentation slots of 20+5 min)

• 3:10pm - 3:30pm: Coffee break

• 3:30pm - 4:45pm: Session 2 (3 presentation slots of 20+5 min)

• 4:45pm - 4:50pm: A 5 min break to rearrange the chairs for discussion

• 4:50pm - 5:50pm: Discussion

Format and activities:

This is a half-day workshop. Participants submit a paper (up to 8 pages). Papers will be reviewed by members of our program committee. Selected participants will have the opportunity to present their papers (20 min presentation and 5 min for questions). Selected participants will be invited to submit their paper to be published as a CEUR On-line Workshop Proceedings (CEUR-WS.org).

Submission:

Papers should be submitted through easychair [link]

Organizers:

• Diego Zapata-Rivera (Educational Testing Service)

• Julita Vassileva (University of Saskatchewan)

New Dates:

• Submission deadline: April 9

• Author notification: April 30

• Camera-ready paper: May 13

• Workshop: June 11 in the afternoon

Program Committee:

• Ivon Arroyo. Worcester Polytechnic Institute

• Ricardo Conejo. University of Malaga

• Vania Dimitrova. University of Leeds (ITS’2018 keynote speaker)

• Sidney D’Mello. University of Colorado Boulder (ITS’2018 keynote speaker)

• Art Graesser. University of Memphis

• G. Tanner Jackson. Educational Testing Service (ITS’2018 Poster & Demo co-chair)

• Irvin R. Katz. Educational Testing Service

• Steve Ritter. Carnegie Learning

References:

[1] Self, J.A. 1999. The distinctive characteristics of intelligent tutoring systems research: ITSs care, precisely, International Journal of Artificial Intelligence in Education, 10, 350–364

[2] du Boulay, B., Avramides, K., Luckin, R., Martinuz-Miron, E., Rebolledo Mendez, G., & Carr, A. 2010. Towards systems that care: a conceptual framework based on Motivation, Metacognition and Affect. International Journal of Artificial Intelligence in Education, 20, 197–229

[3] Kay, J., & McCalla, G. 2003. The careful double vision of self. International Journal of Artificial Intelligence and Education, 13, 1–18

[4] Brusilovsky, P., & Milan, E. 2007. User models for adaptive hypermedia and adaptive educational systems. In P. Brusilovsky, A. Kobsa, & W. Nejdl (Eds.), The adaptive web. Methods and strategies of web personalization. LNCS 4321, Berlin Heidelberg: Springer-Verlag. 3–53

[5] Shute, V. J., & Zapata-Rivera, D. 2012. Adaptive educational systems. In P. Durlach (Ed.), Adaptive technologies for training and education. New York, NY: Cambridge University Press. 7–27

[6] Zapata-Rivera, D. 2017. Toward Caring Assessment Systems. In Adjunct Publication of the 25th Conference on User Modeling, Adaptation and Personalization (UMAP '17), ACM, New York, NY, USA, 97–100. DOI: https://doi.org/10.1145/3099023.3099106

[7] Karpicke, J., & Blunt, J. R. 2011. Retrieval Practice Produces More Learning than Elaborative Studying with Concept Mapping. Science 20 Jan 2011: 1199327 DOI: 10.1126/science.1199327 http://science.sciencemag.org/content/early/2011/01/19/science.1199327

[8] Shermis, M.D., & Burstein, J. 2013. Handbook of Automated Essay Evaluation: Current Applications and New Directions. Routledge Chapman & Hall.

[9] Zapata-Rivera, D., Jackson, T., Liu, L., Bertling, M., Vezzu, M., and Katz, I. R. 2014. Science Inquiry Skills using Trialogues. 12th International conference on Intelligence Tutoring Systems. 625–626.

[10] Graesser, A.C., Dowell, N., & Clewley, D. 2017. Assessing Collaborative Problem Solving Through Conversational Agents. In: von Davier A., Zhu M., Kyllonen P. (eds) Innovative Assessment of Collaboration. Methodology of Educational Measurement and Assessment. Springer, Cham. 65–80