Development of a robot system to support the creation of social capital

Aim of this project

The concept of trust and connection among people in communities such as neighborhoods, workplaces, and schools is called social capital in the social sciences in general. The theory of social capital, in summary, asserts that "society runs efficiently when people socialize moderately and extend loose trust with a spirit of mutual help. In order to alleviate the social problems brought about by the weakening of human relationships and to realize vibrant communities, it is essential to create a system that fosters social capital.

This project will engage in various studies on autonomous interactive robot systems that talk to people in real environments toward the design of social systems that will lead to the creation of new value in fostering social capital. According to political scientist Putnum, face-to-face connections are at the core of fostering social capital. Therefore, systems that support the fostering of social capital must be rooted in activities in physical space. Robots can easily attract people's attention in physical space, and can smoothly carry on a dialogue with multiple people using physical actions such as eye contact and pointing. Through dialogue, robots may be able to promote people's social behavior in the community and make them aware of their connections with others.

Specifically, we are working on the following three themes, aiming to realize a robot system that supports the fostering of social capital by demonstrating the basic technology and its effectiveness in specific situations.

Development of an autonomous interactive robot system that can be easily deployed in real environments
Research on the effects of robot participation in dialogue
Examining the Effect of Praise from Artifacts on Propagation to Others
Demonstration of the effects of robot interaction on human behavior change in real environments

This project places importance on the connection between research results and society. From the perspective of supporting the fostering of social capital, it is also important to enable a diverse range of stakeholders to move robots in the field and collect data. Therefore, Theme 1 does not use expensive robots and sensors that can only be used by large research groups, but builds a system that allows individual researchers and developers to implement the robot's behavior. In Theme 2 and 3, we will examine how to effectively encourage people to change their behavior even with robots that have low interaction ability. And Theme 4 aims to quantitatively clarify how many people actually behave in response to the robot's calls to people in a real environment, and to provide objective data on whether or not the robot system can contribute to supporting the fostering of social capital.

Theme 1: Development of an autonomous interactive robot system that can be easily deployed in real environments

In Theme 1, we worked on the development of a simple and portable autonomous interactive robot system that can be implemented in various situations, rather than a complex and fixed system that can only be implemented in the laboratory. For Sota, one of the commercially available robots often used in human-robot interaction research and general projects, we developed an API for robot control that is not provided by the vendor, and established a method for transferring audio and video from the robot's built-in microphone and camera over a local network. The method was published on Github. Until now, it has not been easy to acquire information from the robot's built-in microphone and camera and input it into a proprietary recognition system, or to control the robot's movements based on the recognition results, but the method we have made available makes it possible. In fact, in the demonstration experiment for Theme 4, the use of this system shortened the development period and facilitated deployment in a real environment. This system is available on Github.

To facilitate the implementation of an autonomous interactive robotic system, we have built a framework for robotic systems based on the concept of microservice architecture, which has recently attracted attention in software engineering. In order to interact with people in a real environment, an extremely complex system must be implemented, consisting of multiple sensors such as 3D distance sensors, microphones, and cameras, as well as multiple recognition modules that perform human orientation measurement, speech recognition, face recognition, gaze recognition, and various types of intention recognition, and multiple action control modules that perform dialog control, speech synthesis, and motion generation, all of which exchange information asynchronously. The system needs to implement an extremely complex system in which multiple modules exchange information asynchronously. In our framework, each module operates as an independent process, and each module is loosely coupled through the mediation of a database (MongoDB) to send and receive messages asynchronously. This facilitates modification and extension of sensors, recognizers, robot action rules, and speech content. This framework is also available on Github, and we are currently organizing the program and preparing explanatory materials.

Theme 2: Research on the effects of robot participation in dialogue

In Theme 2, we showed that even for robots whose dialogue and expressive abilities are inferior to those of humans, the effect of having multiple robots participate in a dialogue on the transformation of human behavior and thinking that leads to support for fostering social capital is more powerful when multiple robots participate in the dialogue. The results are of particular academic value in that they have shown consistent results across a variety of participants (children, adults, and the elderly) and situations (typing practice, reading, learning support, and interview dialogue).

Theme 2-A: Examining the Effects of Multiple Robot Praise on Improving People's Skills

From the perspective of fostering social capital, the act of praising others is essential for building smooth relationships with others. When a person is praised by others, not only does it make the person happy, but it can also have a positive effect on the person himself/herself. For example, it is known that when people are praised by others for their exercise training, they learn exercise skills more efficiently. In this theme, we investigated whether people can successfully acquire motor skills when they are praised by artificial agents (e.g., robots and CG characters) rather than by people. Furthermore, we proceeded to examine the effects of the number of agents and their physicality. The results showed that people who were praised by agents acquired motor skills more efficiently than those who were not praised by agents. The number of agents was increased while the number of praises remained the same, and the change in motor skill was examined in a situation where the participants were praised by more than one agent. In addition, when the difference in the effect of praise was examined between a robot with a physical body and a CG character with a virtual body on a display with regard to the type of agent, human motor skill acquisition was promoted in both cases, and no difference was found in the effect of praise. These results are valuable because they show for the first time that the act of praise, which is essential for supporting SC fostering, can influence a person's behavior even when it is performed by a non-human agent. The results were published in the online journal PLOS ONE (IF=2.74).

Shiomi, M., Okumura, S., Kimoto, M., Iio, T., & Shimohara, K. (2020). Two is better than one: Social rewards from two agents enhance offline improvements in motor skills more than single agent. PloS one, 15(11), e0240622.

Theme 2-B: Examining the effects of multiple robot praise on children's attitudes toward learning

Twin robot that reads picture books to children

Picture book reading is known to have a positive impact on children's language development and vocabulary acquisition. From the perspective of supporting the development of social capital, storytelling is also beneficial for learning and promoting children's social behavior. In previous studies, storytelling has basically been performed by a single robot. In this study, we developed a system in which multiple robots are divided into a reader and a listener, and the listener robot asks questions to the reader robot and agrees to questions from the children to the reader robot. The effectiveness of this system in reading aloud to children was examined in comparison with the conventional reading aloud by a single robot. The results showed that children (ages 3-5) preferred to be read to by a listening robot rather than by a single robot. In addition, the children spoke less during the reading session with two robots, suggesting that the children may have been more engaged in the reading session with two robots. The results were published in the journal Interaction studies (IF=0.824).

Tamura, Y., Shiomi, M., Kimoto, M., Iio, T., Shimohara, K., & Hagita, N. (2021). Robots as an interactive-social medium in storytelling to multiple children. Interaction Studies, 22(1), 110-140.

Twin robot to help children learn English

In addition to reading to children, there has been much development of robot systems to support children's learning. from the perspective of supporting SC fostering, it is important to motivate children to learn in order to help them develop a broader perspective on society. In this study, we focus on English language learning support, which is one of the most active areas of research on learning support systems using robots. Most of these robots are designed for one-on-one learning between a child and a robot. Therefore, in this study, in addition to the robot that mainly provides learning support, we introduced a jury robot that speaks to the child to praise the child according to the child's learning progress. The children's learning time was compared between cases in which the stand-alone learning support robot praised the child and cases in which both the learning support robot and the jury robot praised the child (the number of times of praise was adjusted to be the same). The results showed that the children's learning time was longer when the jury robot participated in the study. This result suggests that praise by multiple robots may increase children's motivation to learn. The results were published in Advanced Robotics (IF=1.699), a leading international journal on robotics.

Shiomi, M., Tamura, Y., Kimoto, M., Iio, T., Akahane-Yamada, R., & Shimohara, K. (2021). Two is better than one: verification of the effect of praise from two robots on pre-school children’s learning time. Advanced Robotics, 35(19), 1132-1144.

Theme 2-C: Verification of a dialogue model using multiple robots to interact with the elderly

Promoting conversation among elderly people or between elderly people and caregivers in elderly care facilities is one of the essential elements to support social capital building. From this perspective, we conducted a demonstration experiment of a conversational robot in a facility for the elderly. In order to promote conversation, it is necessary for the elderly to be able to converse with the robot for a certain period of time as a preliminary step. However, it is extremely difficult to recognize the elderly person's speech. Therefore, it was necessary to develop a technology that would allow the conversation to continue, assuming that speech recognition would not be successful. In this study, we developed a robot-driven question-and-answer dialogue model that avoids dialogue breakdowns by using a dialogue dialogue between two robots' utterances when the speech recognition result is not the expected result. By using this method, we were able to sustain a dialogue for about 14 minutes even in a severe situation where the word recognition error rate of an elderly person's utterance was over 70%. The results were accepted for publication in Applied sciences (IF=2.217), an international journal on applied sciences.

Iio, T., Yoshikawa, Y., Chiba, M., Asami, T., Isoda, Y., & Ishiguro, H. (2020). Twin-robot dialogue system with robustness against speech recognition failure in human-robot dialogue with elderly people. Applied Sciences, 10(4), 1522.

Theme 2-D: Examining the Effectiveness of Apologies by Multiple Robots

When a human being makes a mistake, he/she apologizes to the other person. Since it is inevitable that robots, like humans, also make mistakes, it is necessary to consider how to apologize in order to have the other party accept the apology and forgive. Based on our previous research, we hypothesized that when a robot makes a mistake and another robot joins it in apologizing, people will be more receptive to the robot's apology. We conducted an experiment in which a robot served a customer as a waiter, and when the customer made a mistake by dropping an ordered item, we reproduced a video showing one robot apologizing alone for the mistake and another robot apologizing together with the two robots, and collected questionnaires on the web. The results of the experiment showed that when the robot that made the mistake apologized, the apology was accepted more when two robots apologized together than when one robot apologized alone. In addition, trust in the robots increased, and dissatisfaction with the store providing the product decreased. These results indicate that the apology is accepted more when the number of robots apologizing is increased from one to two. This result indicates that even though the robot is an artifact, the apology is more acceptable when more than one robot is used to apologize. From the perspective of fostering social capital, this study can contribute to the design of behavior when one robot supports another robot that has made a mistake, just as a human clerk supports a colleague who has made a mistake. The results were published in the American scientific journal PLOS ONE.

Okada, Y., Kimoto, M., Iio, T., Shimohara, K., & Shiomi, M. (2023). Two is better than one: Apologies from two robots are preferred. Plos one, 18(2), e0281604.

Theme 3: Examining the Effect of Praise from Artifacts on Propagation to Others

In fostering social capital, it is a very important question whether cognitive and behavioral changes that occur in humans as a result of human-robot interactions propagate to others. If a change in a person's behavior caused by a robot also affects the person's interactions with others, the effect of the robot will spread widely in the community. Based on the results of our previous work on human behavior change by praise, we proceeded to examine how people who were praised or encouraged by the robot would behave toward others afterward. In the experiment, participants were given a simple task on a PC. Here, participants experienced the robot either praising, giving only neutral information, or agitating. In addition, the same participants were given the additional task of evaluating other participants' assignments. In reality, the other participant did not exist, and the performance of the task was a pre-recorded and replayed version of his/her own task. In other words, the participant was able to give praise and incentive to the virtual participant who performed the task with the same performance as he or she had received from the robot. The results of the experiment showed that participants who were praised by the robot praised other participants more, while those who were fanned by the robot stopped praising other participants. This result indicates that the social attitudes of the robot, an artifact, propagate to the social attitudes of the other participants who were praised. In other words, the results suggest that the robot may be able to contribute to fostering social capital by creating propagation and circulation of praise starting from the robot. These results were accepted for publication in the International Journal of Social Robotics, one of the leading journals in the field of robotics.

Higashino, K., Kimoto, M., Iio, T., Shimohara, K., & Shiomi, M. (2023). Is Politeness Better than Impoliteness? Comparisons of Robot's Encouragement Effects Toward Performance, Moods, and Propagation. International Journal of Social Robotics, 1-13.

Theme 4: Demonstration of the effects of robot interaction on human behavior change in real environments

In Theme 4, we will use the robot system developed in Theme 1 to clarify the effects of interactions from the robot on people's behavior change in various situations where people actually live. From the perspective of supporting the fostering of social capital, we are examining the impact of robots on society by operating the robot system in real environments, working to improve the technology, and collecting and analyzing real people's reactions.

Theme 4-A: Demonstration Experiment on a Robot to Explain Exhibits at a Science Museum

We conducted a demonstration experiment in cooperation with the Tsukuba Expo Center, a science museum in Tsukuba City, Ibaraki Prefecture, to confirm that the robot system developed in Theme 1 works properly in a real environment and to see how well the robot can attract people's attention. Specifically, a robot was deployed in front of an exhibit at the science museum to interactively explain the exhibit, and the behavior of visitors during the exhibit visit was verified depending on whether the robot was present or not. The results of the analysis showed that 38 participants (32 adults and 6 children) visited the exhibit with the robot (3 hours and 20 minutes) and 167 participants (112 adults and 55 children) visited the exhibit without the robot (6 hours). The proportion of visitors who completed the tour in a very short period of time (less than 30 seconds) decreased by 13.5 percentage points when the robot was present, increased by 11.1 percentage points for visitors who visited for 30 seconds to 1 minute, and increased by 7.2 percentage points for visitors who visited for 2 minutes to 2 minutes and 30 seconds, compared to those who did not have a robot. In other words, the results suggest that the robot's explanation may increase the duration of the tour. If multiple robots were introduced and deployed in exhibits that are not usually explained by humans, it is expected that the total time spent visiting exhibits in the science museum would increase significantly.

Theme 4-B: Demonstration experiment on a robot that asks people to answer a questionnaire on a university campus

In the Theme 4-A experiment, there was a small but significant incentive for people to interact with the robot, namely, the ability to listen to the explanation of the exhibit. Even in situations where there is little benefit to the person, it would be very meaningful from the perspective of supporting the cultivation of social capital if a specific action could be promoted by the robot's voice. Based on this idea, we considered asking people passing by in a real environment to respond to a questionnaire using a robot. Specifically, we placed a robot and a tablet terminal for answering a questionnaire in front of the University of Tsukuba cafeteria, and had the robot remotely control the robot to ask passersby to answer the questionnaire. As a result of testing whether there was a difference in the survey response rate between days when the robot was placed and days when it was not, the survey response rate was higher on the day when the robot was placed and on the day after the survey was changed. After that, however, the response rate did not change much whether the robot was present or not. We believe that this is because the same people passed by the robot every day, and the effect of the robot's voice call was lost due to familiarity with the environment. However, the fact that the robot's voice call improved the response rate, even if only temporarily, gives us hope for the effectiveness of the robot's support in fostering social capital. The results of this study have been accepted to Ro-man2020, a major international conference on human-robot interaction.

Natori, T., & Iio, T. (2021, August). An empirical study of how much a social robot increases the rate of valid responses in a questionnaire survey. In 2021 30th IEEE International Conference on Robot & Human Interactive Communication (RO-MAN) (pp. 951-956). IEEE.

Theme 4-C: Demonstration of questionnaire collection at the Expo Center

Theme 4-C showed that with the current robot system, the impact of the robot's voice call drops off in a short period of time due to people's habituation. On the other hand, if many of the visitors to a science museum are visiting for the first time or for the first time in a long time, the current robot system may have a certain effect. Therefore, we conducted a demonstration experiment at a science museum (Tsukuba Expo Center) that extended Problem 3-2. Specifically, we made the robot autonomous and compared the survey response rate under three conditions: when the robot requests a survey response, when the speaker requests a survey response, and when the robot does nothing. After 7 days of measurement under each condition, the results showed that when the robot requests a survey response, the survey response rate increased approximately 1.7-fold compared to the voice-only case, and approximately 3-fold compared to the case where the robot does not call out. In addition, a follow-up test was conducted in a different location within the science museum, using only the robot and speaker conditions, and the response rate increased approximately 2.5 times more in the robot condition than in the speaker condition. These results suggest the effectiveness of prompting survey responses by a robot or speaker, even without a human call. In particular, it is clear that robots are more effective in prompting passersby to respond to a survey than voice alone. Although the percentage of total passersby may not be that large, the robot or speaker can be installed and left alone, making it cost-effective and effective.

Theme 4-D: Examining the Effects of Conversations with Robots on College Students' Mood

Maintaining the mental health of college students is a major issue. In surveys of university students taking leaves of absence, dropping out, and staying in college, a certain number of students are found to have taken leaves of absence due to mental disorders. In terms of supporting the development of social capital within universities, it is highly important to provide preventive interventions before students' mental health problems worsen. Therefore, we aim to examine the feasibility of a robot that maintains students' mental health by investigating the content of dialogues that contribute to improving their moods as part of this study. Specifically, we conducted an experiment to examine the degree to which two types of dialogue, with and without praise, improve the mood of students on the street in a real environment on a university campus. A 16-day field trial was conducted in a real environment using the WoZ method. In the experiment, two conditions were set up regarding the content of the conversation: a praise condition and a no-praise condition. The results showed that participants who experienced the praise condition significantly improved their mood (especially confusion and fatigue) before and after the conversation compared to those who experienced the no-praise condition. This suggests that this is one of the possibilities for a meaningful mental health intervention within the university.

Future Developments

The results of this project have the potential to influence and transform the flow of research in multiple fields, including robotics, artificial intelligence, social sciences, and information media. In robotics, the project will create a trend from intelligent robotics research that engages with people to social robotics research that connects people with people, and the dialogue data obtained from social implementation will promote the development of artificial intelligence. In social science, it will provide a testbed system for hypothesis testing and enable a variety of intervention experiments. In information media, it is expected to generate new design theories based on an understanding of the connections in physical space. In addition, the results of this research can be deployed to mitigate problems caused by the lack of social capital in the community. For example, it could bring new solutions to the problem of bullying in elementary and junior high schools. While direct instruction to perpetrators and bystanders of bullying often leads to backlash, the results of this research can spread the voices of victims within the community through robots and make perpetrators and bystanders aware of their potential connection to the victim. The chain and cumulation of such awareness will change the structure of the community to one in which bullying is more likely to be discouraged. In addition, conventional robotic communication support for the elderly is closed to the relationship between the robot and the elderly and does not connect the relationship with the community. The results of this research are expected to activate communication among the elderly and between the elderly and caregivers, resulting in improved health for the elderly in nursing homes.

We believe that this research, which deals with the transformation of human cognition and behavior through human-robot interaction in physical space, will contribute to the realization of a truly "human-centered society" that goes one step beyond the traditional "substitution of human tasks by AI and robots" to foster trust and connection with people in the community.

Google Sites

Report abuse