1 Introduction

Individuals with communication difficulties include the ones who are apprehensive about communication and encounter several problems in their daily online interactions [1–3]; such as limited right to talk (RoT), lack or absence of social support (SS), and low social presence (SP) in communication. The first stated problem, RoT, is defined as an individual’s feelings concerning the provision of equitable speaking and opinion-expressing opportunities relative to their peers in conversation. Such feelings are influenced by different types of elements in their conversations, such as the number of conversational turns, duration of listening and talking [4], number of utterances, inter-utterance pauses, and back-channel responses [5]. Ideally, equality is required among all types of elements for each conversation peer as general rules of communication. However, in reality, such rules are severely prone to violations that trigger vocal interruptions, speech disruptions, and social anxiety among peers [6]. The risk of violating these rules can be significantly minimized if conversation peers socially support each other. The second aforementioned problem, lack or absence of SS, is defined as information that leads the subject to believe that he is cared for, loved, esteemed, and considered a member of a network with mutual obligations [7]. In conversations, the presence of SS increases the willingness to communicate with individuals [8] and minimizes anxiety and depression [9]. Finally, the third aforementioned problem, low SP, is defined as an individual’s perception of their presence in a conversation [10]. It contributes to the feelings of belonging and connectedness in a conversation [11,12] Further, such feelings influence an individual’s sense of being attended to (SoBA), which is defined as the experience of receiving appropriate focus and attention in a conversation, including being listened to and answered [13].

Computer-mediated communication (CMC) technologies have been investigated to resolve the challenges of limited RoT, lack or absence of SS, and low SP issues for individuals with communication difficulties. CMC includes text, audio, and video interaction technologies. Although each CMC technology provides a huge support to enable people with communication difficulties establish effective communication with others, these technologies fail to assure the RoT of users owing to their limited ability to transfer verbal and non-verbal information that influences the adherence to communication rules [14]; hence, users need to reconcile RoT via mutual cooperation [15,16]. Furthermore, CMC technologies are limited in offering SS to users [17]; for example, text-only technology is significantly limited in offering SS [18]. However, audio-only technology can provide limited SS [19], because in addition to verbal cues, it also provides limited non-verbal cues, such as vocal tone variation associated with discrete emotions [20]. In contrast, video technology is relatively better at providing SS to users [21] because, in addition to verbal cues, it also provides limited non-verbal cues such as awareness of attentional focus, ease in speaking turns, and facial expressions [22,23]. Apart from SS, CMC technologies are limited in providing a SP to users [24]. Text-only technology provides a severely low SP to users [24,25], while audio-only technology provides a low, but better SP than text-only technology [26]. Although video technology is relatively robust in providing SP to users [27,28], the presence of video streaming causes an increase in the communication apprehension of users, which eventually decreases the ease of talking [29–31]. To prevent such problems while maintaining SP, we must explore other available technologies. Considering the performance of text- and audio-only technologies regarding the provision of SP to users, they are also expected to be severely limited in providing SoBA to users. However, video technology can provide a significant amount of SoBA to users, as it provides a rich social presence [32]. In conclusion, text, audio, and video technologies are all limited in their ability to address the aforementioned problems.

Robot avatar technology has been investigated considering SP; however, the RoT and SS challenges in communication are yet to be studied. There are two types of robot-avatar technology users: those interacting through avatars (hereafter referred to as operators) and those interacting with avatars (hereafter referred to as visitors). In social interactions, using a single physical avatar facilitates communication between operators and visitors, such as an interaction between a teacher and a student in an educational environment [33,34] and an interaction between two family members [35,36]. However, such interactions are dyadic in nature; because on the visitor’s side, only a single physical avatar and a visitor interact with each other, while on the operator’s side only an operator and a visitor interact with each other. In other words, the ratio of amount of talk of each person needs to be 1/2 in dyadic interaction i.e., equal number of utterances, equal number of words, and equal inter-utterance pauses [37]. However, based on the personality of the person, such an aspect varies from person to person, so it is difficult to increase the operator’s RoT; especially when he/she has communication apprehension. Therefore, this raises the following research questions: In an avatar-mediated communication, is there any way to manipulate feelings of RoT in operators? Such a question is yet to be explored. Experiences via avatars are considered the operators’ experiences [38], and when communicating with a visitor via an avatar, the operator cannot avail SS from their own robot avatar. Because a dyadic interaction exists with the visitor via the avatar, the operator can only avail limited SS. The feelings of receiving SS are crucial in interactions, especially for people with communication difficulties [39]; therefore, alternative methods are required. A human subject can experience a high SoBA by watching a video scene of a conversation, where a visitor interacts with the avatar of a side-participant [13]. This implies that an operator feels supported in conversation via the avatar when another avatar cares for their avatar. The usage of two avatars is useful from visitors’ point of view e.g., 1) producing coherence in conversation [40], 2) improving social behaviors [41], 3) generating pressure in communication [42] and, 4) persuading for certain actions [43]. Now, it is important to investigate the effect of adopting a second avatar on the operator communicating via an avatar. The visitor talking with the two avatars of the operator would be engaged in a triadic interaction; where, as per triadic interaction rules of talk, the ratio of amount of talk for each one needs to be 1/3 of total amount of talk [37]. As both robots would be the avatars of an operator so, it is expected that operator’s RoT will increase according to the rule of communication for triadic interaction. Moreover, there is a possibility of including SS for an operator from avatars in such triadic interaction. Therefore, in this study, we propose a novel system comprising two avatars and controlled by an operator, to communicate with the visitor and experience higher RoT and SS.

In the proposed system (refer to Fig 1), the operator’s utterances emerge from either of the teleoperated avatars. The choice of the speaking avatar and the production of backchannel responses from the other avatar were processed randomly. The frequency of switching avatars was depending on how often an operator wants to utter; where switching of avatars was triggered by typing or speaking of the operator. Similarly, the timings of switching of avatars was based on the operator’s speed of preparing utterances. Sometimes, the second avatar took a speaking turn from the current avatar to talk about the same opinion as the first avatar; as the speaking content is defined by operator only. Such turn-taking and backchanneling behaviors of the second avatar are expected to make the operator feel supported in communication. Consequently, the visitor also had to switch attention towards the speaking robot avatar throughout the conversation. Such approach by the visitor was evident to the operator by the video feed on the monitor. Furthermore, the proposed system is expected to provide another advantage of increased RoT because an operator is expected to attend a multiparty conversation with two agents. Humans usually tend to equalize conversational turns, time to listen to and talk [4], including the number of utterances, inter-utterance pauses, and back channel responses in conversation [5]. This indicates that humans tend to expect an equal RoT for each participant during the conversation. Because experiences via the avatars are considered the operator’s own experiences [38], the assigned RoT of each avatar is expected to be perceived by the operator as their own RoT in a multiparty conversation. Therefore, in the proposed system, the operator is expected to perceive a maximum of double RoT levels. Conversely, at least more than one is expected to be perceived when communicating via a single avatar.

Download:

• PNG
  larger image
• TIFF
  original image

Fig 1. Proposed robotic system with two teleoperated robot avatars.

https://doi.org/10.1371/journal.pone.0292803.g001

2 Video teleconferencing system involving physical avatars

A schematic of the proposed system is presented in Fig 1. It comprises a computer, a monitor screen, a headset with a microphone, a keyboard, two semi-humanoid robots, and a web camera. The components used to develop the proposed system were ordinary that were commercially available easily. Using a computer and web camera, an online interaction session was arranged between the operator and visitor physically present at different locations (Locations I and II). The robots were physically present at Location II, one in front and another one beside the visitor. Both robots were avatars of the operator. We utilized the CommU robot, which was developed via collaboration between Osaka University and Vstone Co., Ltd., Japan [44]. The CommU robot is a semi-humanoid robot with clear eyes of 14 degrees of freedom in total and a 31-cm height; in addition, it is programmable using the JavaScript language and can interact via visual, speech, and motion stimuli.

To control the robot avatars over a wide area network in real time, a locally built graphical user interface (GUI) was adopted by the operators, as illustrated in Fig 2. The GUI comprises two sections: Sections I and II with visual feedback and utterance-related handling-options, respectively. The visual feedback section was designed using web-RTC, which provides a real-time view of the visitor’s environment to the operator, referred to as Section I in Fig 2. It also displays the detected spoken answers of the operator in the middle of the section as dynamically added buttons at the bottom; where to detect and to convert the spoken answers into text, a Web speech API, written in java script programming language, was used. However, the utterance section comprises a text field with three buttons, referred to as Section II in Fig 2. It provides several facilities to the operator, such as typing new answers, editing previously detected answers, deleting answers, and enabling/disabling speech recognition.

Download:

• PNG
  larger image
• TIFF
  original image

Fig 2. GUI on the operator’s computer screen.

https://doi.org/10.1371/journal.pone.0292803.g002

In an online interaction session, the proposed system behaves in two different ways in which an operator provides the answer(s) to the visitor’s question(s). In a case where the operator does not provide the answer(s), Avatar-I remains still with a visual focus toward the visitor, while Avatar-II continues to perform idling motions, i.e., the avatar keeps switching visual focus between the visitor and Avatar-I by turning its head and torso. However, in a case where the operator provides the answer(s), one of the avatars utters in a synthesized voice in front of the visitor while raising their left arm. Meanwhile, the other avatar shifts its visual focus to the uttering avatar and acknowledges them by nodding, thus pretending that the provided answer is accurate and acceptable. Then, the system randomly chooses an avatar to deliver the operator’s answer. While conversing via such a system, the visitor asks questions from the avatar who delivers the answer and the operator is required to answer the questions either by speaking or typing. The proposed system also manages conversational turns between the operator and visitor. When the visitor speaks, the operator obtains information from a real-time video feedback. However, when the operator types or speaks the answer, the visitor obtains information from the glowing (red) cheeks of the robot(s).

3 Materials and methods

4 Results

4.3 RoT

The Wilcoxon-signed rank test was conducted to identify the effect of the condition adopted (single avatar vs. double avatar) on the operator’s RoT feelings. The median value of the RoT for the operator of the double-avatar condition (Mdn = 35) was significantly higher than that for the single avatar condition (Mdn = 34) (Z = -1.99, p = 0.047, r = 0.23), as presented in Fig 3.

Download:

• PNG
  larger image
• TIFF
  original image

Fig 3. RoT of operators.

https://doi.org/10.1371/journal.pone.0292803.g003

4.4 Perceived SS

The Wilcoxon-signed rank test was conducted to identify the effect of the condition adopted (single avatar vs. double avatar) on the SS of the operator. It was demonstrated that the median value of the SS for the operator of the double-avatar condition (Mdn = 15) was significantly higher than that for the single avatar condition (Mdn = 15); (Z = -2.11, p = 0.034, r = 0.24), as presented in Fig 4.

Download:

• PNG
  larger image
• TIFF
  original image

Fig 4. SS of operators.

https://doi.org/10.1371/journal.pone.0292803.g004

4.5 SoBA

Two paired-sample t-tests were conducted to identify the effect of the type of condition adopted (single avatar vs. double avatar) on the SoBA of the operator. The mean value of the experienced SoBA for the double-avatar condition (M = 18.29, SD = 4.26) was not significantly higher than that for the single avatar condition (M = 18.13, SD = 4.06), (t (36) = -0.33, p = 0.74, d = 0.054), as illustrated in Fig 5.

Download:

• PNG
  larger image
• TIFF
  original image

Fig 5. SoBA of operators.

https://doi.org/10.1371/journal.pone.0292803.g005

5 Discussion

Experimental results demonstrated that operators talking via double avatars experienced relatively higher communication support compared to those talking through a single avatar, and communication support was assessed by RoT, perceived SS, and SoBA indices. Operators with double-robot avatars perceived significantly higher SS in communication compared to operators with a single-robot avatar. Similarly, operators with double-robot avatars felt significantly more RoT in conversation compared to operators with a single-robot avatar. However, there was no significant difference between the SoBA of operators using single- or double-robot avatars. It is inferred that the use of double avatars provides communication support to operators by manipulating their feelings of RoT and increasing their perception of SS in conversations.

The primary reason for the significant increase in RoT was the operator’s feeling of owing the individual RoTs of their two remote representations i.e., their avatar agents. Humans tend to follow the rules of conversation [4,5]. In our conversation setup in which a visitor interacts with two avatars simultaneously, the conversation is perceived as a triadic interaction scenario by the operators. In such a triadic interaction scenario, an equal RoT is received by each peer of the conversation, i.e., the visitor and avatars. However, because the operators teleoperated the avatars, and experiences via avatars are considered their own experiences [38], the individual RoT of each avatar would eventually become the operator’s RoT; the main cause of the observed effect significantly increased the RoT feelings of the operators. Such an accumulative RoT would have a maximum of up to two-fold. An important point to keep in mind for obtaining the less effect of RoT for operators than the maximum i.e., two-fold could be explained by the varying degree to which an operator accepts the individual RoT of each avatar as his/her own RoT. The primary reason for the significant increase in SS was the operator’s feeling of being supported by avatar agents in front of the visitor throughout the conversation, even when there was a difference in opinions. SS refers to the information transferred to a subject that leads them to believe that they are cared for, loved, esteemed, and considered a member of a network with mutual obligations [7]. How and in which specific time frames such information is being transferred are crucial elements influencing the beliefs of subjects [49]. In general, such information can be transferred to the subject by exhibiting verbal, non-verbal, or both reactions, i.e., verbal and non-verbal behaviors in a specific time frame. Here, the robot avatars transferred such information by exhibiting non-verbal supportive behaviors to the operators in front of the visitors, specifically at the time when they finished expressing their opinions. Such supportive behaviors were displayed to the operators throughout the conversation session, which was the major cause of the observed effect, i.e., the significantly high perceived SS.

Operators talking via double avatars were expected to have a significantly higher SoBA compared to operators talking through a single avatar. However, this effect was not observed in this study. Instead, a negligible increase in the SoBA was observed. Two possible reasons can explain the absence of such an effect on the operators: 1) differences in the treatments of the visitors, and 2) the effect of partial occlusion in the field of view for the operator. In remote interactions, SoBA is more related to the treatment of visitors towards the operator’s remote representations i.e., avatar robot(s). Such treatments were expected to be perceived by operators as treatment for themselves. However, as the visitors differ from each other in terms of reproducing similar behaviors as instructed, so it might be possible that the operators did not receive similar types of treatments. Therefore, it is quite natural to accept such a fact because the degree to which the given instructions are strictly followed varies among visitors i.e., humans [50–52]. Moreover, one of the avatars was placed in the line of sight of the operator i.e., in front of the camera in such a way that the rear side of the head and torso was evident, thereby triggering a small degree of visual occlusion in the field of view of the operator. For operators, such a visual occlusion hinders the process of direct visual attention from visitors and eventually influences the quality of interaction [53]. Therefore, the presence or absence of direct visual attention from visitors influences the SoBA of operators [13].

Despite communication support via double physical avatars, a few limitations remain. First, the degree to which each participant perceives a lack of SS and RoT in daily life communication was not controlled. Furthermore, the degree to which each participant prefers a specific type of SS was not controlled; as it might be possible that one participant prefers verbal SS over non-verbal and vice versa. Similarly, the degree to which each participant prefers a specific timing for receiving SS was also not controlled; as it might be possible that one participant prefers to receive the SS immediately after the end of the utterance than receiving it after a delay of certain seconds. Second, we did not recruit participants with a severe lack of perceived SS and RoT communication issues. Third, all the materials of the experiment were translated into Japanese, with a specific linguistic and cultural background. Therefore, the observed effects do not necessarily guarantee real-world reproducibility. The obtained minimal significant results are limited as we hired the participants having no severe lack of perceived SS and RoT issues in their daily life communication. Therefore, interactive experiments with individuals having a severe lack of perceived SS and RoT issues using the proposed system in a more controlled manner are required to observe the actual potential of the system in practice and draw more decisive conclusions. The observed effect of SS was significant. However, the average values were the same for both clearly distinctive experimental conditions. The possible reason could be that a few of the participants virtually recognized both conditions as dyadic in nature; as apparently, there were only two humans involved in communication alongside a third agent that wasn’t a human but a robot avatar.

Moreover, in switching avatars, since the utterances produced were opinions of operators with a predefined non-verbal social behavior so, there was an absence of additional emotional responses from the third agent. It might be a difficult situation for operators to mentalize the agent. The introduction of a second operator for controlling the third agent could be a possible solution for: 1) decreasing the possibility for operators to virtually recognize both conditions as dyadic in nature and, 2) mentalizing the third agent. In future interactive experiments, consideration of such a factor would also help to gauge the potential of the proposed system. The differences in the personalities of the individuals with respect to the presence or absence of a severe lack of perceived SS and RoT will require defining a proper criterion with the help of a professional e.g., psychiatric or psychologist to categorize the individuals with minor, moderate, and severe lack of perceived SS and RoT. However, such a categorization process will require further preliminary experiments; as there will be a need for modifications in the questionnaires of perceived SS and RoT scales to cover the context of lack of perceived SS and RoT in daily life communication.

Another major limitation of our study is the adoption of the non-verbal behaviors of avatar robots to influence the perceived SS of operators. We did not explore the effects of using verbal or a combination of verbal and nonverbal behaviors of avatar robots on the perceived SS of operators. Moreover, we did not explore the effect of the time of SS provision to operators. Furthermore, we did not verify whether the effect of RoT will continue to increase as the number of avatar robots increase. For simplicity, we focused solely on the effects on the operator’s side. However, the use of the proposed system in real life would also require acceptance from visitors, as their chances of being affected by the operators’ severe lack of SS and RoT in communication are higher.

A part from limitations, we also interviewed the participants at the end of experiment; where they were required to share their positive and negative experiences with us so that we can further explore the potential of the proposed system. A few positive comments are as follows: 1) The voice recognition and utterance features were nice. 2) Two different anonymous voices for each avatar was a nice feature; as it ensured the protection of the personality behind the second avatar. 3) Although the discussion was difficult, however, I liked it. The system was good for heated discussions; where I could say opinions more openly without compromising friendships and showing actual emotions to others. 4) The feature of being anonymous is useful in real-life applications. 5) In case we are busy, the system has the potential to support us. 6) Such a system would help a lot in reducing the shyness in the conversation. 7) The system seems great. I think it will help a younger sportsman to talk more openly with the captain of a team. Similarly, a few negative comments are as follows: 1) Two robot condition was a little bit difficult because the shifting of gaze of the visitor was random, depending on the uttering robot. 2) The switching of robots was a little bit confusing for me. 3) If we can choose the robot to speak, I think, it will further improve the system. 4) Answering the long sentence made the usage of the system difficult for me. 5) One robot condition was easier than two. 6) While typing or speaking, the thoughts about, which robot is going to speak next? are confusing. 7) If I know which robot is going to speak next, I will be more relaxed.

In addition to these limitations and comments, some challenges would hinder the integration of the proposed system into daily life. In the beginning, it might be challenging to find appropriate individuals with severe lack of perceived SS and RoT issues in communication, and later train them to use such a system in daily life independently. Furthermore, in subsequent stages, it might also be challenging to endure the cost of deploying the system and later bear the maintenance cost together with multiple unforeseen technical and non-technical issues, for which individuals with a severe lack of perceived SS and RoT will be completely dependent on service providers. Despite of aforementioned challenges, the improvement of the lack of perceived SS and RoT should remain the top priority to increase the quality of daily life communication of individuals. To ensure such a priority, the involvement of professionals e.g., psychologists or psychiatrics would be necessary.

6 Conclusion

Here, we demonstrated that a robotic video conference system with two teleoperated robot avatars significantly increased the operator’s feelings of RoT and SS in online conversations. While having remote experiences via both robot avatars simultaneously, the operator could speak through any of the robot avatars. Because talking via two teleoperated robot avatars will eventually become a triadic interaction scenario on the visitor’s side, they will be required to abide by the rules of triadic conversation. Such a situation is advantageous for the operators because the individual RoT of each robot avatar will ultimately become the operator’s RoT. Similarly, the SS provided by the robots to each other will also become SS to the operator in front of the visitor throughout the conversation. Moreover, the operator’s SoBA is also expected to increase, as the visitor has to be more attentive to the operator via robot avatars. To verify such expected effects, a field experiment was conducted with RoT, SS, and SoBA as the measured indices. The obtained experimental results indicate the positive effect of using two avatars on the operator’s RoT and SS, but not on the SoBA. In the future, we will examine the effects of using the proposed system for people with a severe lack of perceived SS and RoT issues in communication.

Supporting information