Research

Future Worlds is an educational resource for science museums and K-12 science classrooms, as well as a research platform for investigating AI-driven learning technologies in formal and informal settings. A series of pilot studies held in the North Carolina Museum of Natural Sciences has shown that Future Worlds is highly engaging for learners as they explore sustainability concepts and solve simulated environmental problems. To assess participants’ sustainability understanding before and after exploring Future Worlds, several complementary measures have been administered during the project’s museum studies, including two novel instruments created by the research team.

Personal Meaning Maps. Personal meaning maps (PMMs) consist of a blank piece of paper with a brief set of instructions and a prompt phrase, in our case sustainability. Participants used a blue ink pen to write or draw words, phrases, and pictures about what they thought and knew about the prompt word on the blank paper. After the learning experience, participants were given the opportunity to revise their PMM using a red ink pen. After the study, two raters scored each PMM based on the relevance and accuracy of each element included on the page. The raters summed the total number of relevant and correct items listed and subtracted the total number of irrelevant and incorrect items included. The inter-rater reliability for the pre (r = .84) and post (r = .88) versions achieved acceptable levels.

Identification Task. Learners were instructed to inspect an illustrated picture of an environment—which depicted both sustainable and unsustainable environmental practices—and annotate the picture by circling “good” environmental practices and crossing out “bad” practices. Learners’ pictures were returned after exploring Future Worlds, and participants revised their annotations. Two independent raters scored learners’ annotations by using a rubric vetted by subject matter experts. The raters obtained high scoring agreement on both the pre-test (r = .97) and post-test (r = .95) versions.

Image Sorting Task. Learners were given paper copies of ten images depicting various environmental practices (e.g., recycling, solar panels, heavy traffic congestion, off-shore drilling). Participants were asked to organize the images into two categories of their choosing and were instructed to select the two categories such that they contained as similar a number of images as possible. An expert-based categorization of the images into “sustainable” and “unsustainable” categories was considered the gold standard response, and this benchmark was used to grade students’ responses. This activity was completed both before and after exploring Future Worlds.

In addition to data on learners’ conceptual understanding of sustainability, several other sources of multimodal data have been collected during Future Worlds studies, including facial expression, body movement, conversation, and eye tracking data. This data has been analyzed to investigate dwell time, learner behavior, and produce transcripts of participants’ conversational behaviors. Fine-grained trace logs of learners’ multi-touch interactions with the Future Worlds software have also been gathered for subsequent learning analytics.

If you would like to learn more and/or gain access to the software platform that we developed to investigate visitor engagement with Future Worlds using multimodal learning analytics techniques, please contact us at intellimedia@ncsu.edu.