Brendan O'Connor - rateMyChat

Rate My Chat

Brendan O'Connor

Music and Speech Modelling 2019

Abstract

This project explores multiple behaviours of speech prosody. Using a combination of Logic, Praat and Matlab, prosodic elements are collected from a sample of dyadic conversations. This data is analysed so that the behaviour of these prosodic elements are split into multiple variables, some of which take one or both conversing participants into consideration. These conversations are also classified by a rating of 'conversational quality'. The behaviour of these variables were expected to correlate with the conversation ratings which was true in many cases. Further exploration highlights areas for future research on this topic, and how its focus can be tightened to extract more accurate data.

1 INTRODUCTION

Conversation can be a challenging activity for some. For others, it takes as much effort to verbally interact with people as it does to breath. Whether this is down to being an extrovert, introvert, unfamiliarity with the topic or person, or just discomfort, it can be presumed that the 'successful' structure and flow of a conversation is not only determined by the speaking participants, but also by a number of psychological factors influenced by their environments and nature of engagement.

It is fascinating to hypothesise how exactly these factors manifest themselves within the voices of those conversing. I intend to investigate this by analysing the behaviour of variables related to speech prosody such as pitch, tempo and amplitude. Other variables include those related to conversational flow, such as silences, overlapping and backchannels. Linguistics and semantics are entirely ignored and their analysis is considered outside the scope of this study.

The definition of a good quality conversation can be extremely ambitious. Therefore, it is important to first discuss why and how a conversation may be perceived as good or bad. This also depends on the values of the classifier, which in this case is the author. The definition of a good-quality conversation will therefore be based on the following conditions:

Both speakers must:

• Feel that they are being listened to

• Have equal floor-time

• Provide significant information 

• Provide their opinions

• Display some interest in the topic

Therefore in order to carry out this study, it is important that we consider how to detect these conversational conditions within the voice.

It is clear from previous literature that there should be more than enough information in vocal cues to estimate parameters of involvement and emotional investment. However, the prediction of the quality of entire conversations based on these specifications is the primary interest in this study. To be able to do something so general on a dynamic and fluctuating system could prove to have excellent applications. The automatic evaluation of conversations would lead to vast amounts of discovery and improvement in academic, industrial and personal environments. It will also be interesting to consider why some vocal attributes will correlate with the classification while others do not.

2 RELATED WORK

An abundance of research has been conducted regarding the detection and prediction of emotion in speech [11] [4] [3], as well as analyses of conversational structure [13] [9] [6]. The behaviour of vocal cues have also been studied in relation to structural aspects of conversation such as persuasion [8], interruption [17], assurance [7] etc. This section briefly touches on interesting findings that have highlighted hypotheses worth considering when shaping this study.

It should be noted that the definition for a good conversation described above does not consider valence of speakers. [1] discusses the value of argumentativeness in a conversation, describing it as constructive trait while verbal aggressiveness may be considered a destructive one. This allows participants to discuss controversial topics and even counteract the opinion of another [10]. Therefore negative emotions can also be attributed to good conversation, which informs this study to reconsider searching for valences of emotions.

Valence in any case, is far more difficult to detect than arousal, as [14] have successfully tested while trying to determine suitable methods of vocal feature-extraction and emotion detection. They also concluded that jitter and shimmer are strong prosodic features that correlate heavily with successful emotion classifications.

For a speaker to feel listened to, it is can be assumed that one of the aural ways this feedback can be conveyed is through back-channeling. [16] confirmed this by observing that the inclusion of consistent back-channeling caused their interview-based conversations to be elongated by 31%. The nature of back-channeling: How it is produced; received and how it affects the conversation; was observed by [19], who’s research has informed the study in designing appropriate algorithms to detect backchannels. An interesting question relating to data collection is what affect different genders, cultures and languages have on the tells of a good conversation. While trying to observe the relationship between emotion and pitch, [18] discovered that different genders and languages in relation to average pitch, pitch range and jitter caused negligible differences. This will be challenged in the Discussion section.

The nature of interruptions was analysed by [17]. They examined the difference and relationship between listener’s and speaker's amplitude before, during and after interruptions. Interestingly, they found a participant's average vocal volume has no influence over whether their interruption will be successful or not, but an increase in their amplitude will increase chances of a successful interruption if the participant is the defender.

The behaviour of silences and speech within a conversation were studied by [15]. They learned that speech and silence chunks occurred in small fragments rather than long durations, and have proven that the amount of small silences in between spoken words and sentences during a speakers turn amounts to a great deal, leading to a significantly more balanced ratio than expected between silence and speech. This is evident in the results of this study.

3 METHOD

Data set

The data set classified and analysed in this study is the Switchboard-1 Release 2 edition (Godfrey, John, and Edward Holliman. Switchboard-1 Release 2 LDC97S62. Web Download. Philadelphia: Linguistic Data Consortium, 1993) provided by the Linguistics Data Consortium (LDC). It consists of dyadic conversations between both males and females from Texas. Both participants are native English speakers and strangers to one another. Conversations are recorded at a sample rate of 8kHz, and recordings are claimed to be separated into two channels. Conversation topics are assigned to participants that have chosen them (up to 50 out of 70 topics can be chosen by any participant). No two people ever conversed more than once and topics were only used once per participant.

The restrictive nature and limiting conditions of this data set make it the ideal choice for this study. The importance of using phone-call conversations was made clear by the discoveries made by Kraus [12], who discovered that emotion was translated better through vocal cues only, instead of visual or audio-visual stimuli. It was confirmed that prosodic behaviour carried with it a great deal of emotional information, and when the listener only has aural cues, greater empathy for the speakers emotion is achieved. The voice-only condition enforced by the nature of the data set was not chosen to influence the participants, but rather to provide the classifier (the author) with maximum information and insight into the quality of the conversation in order to make the most accurate classification.

Classification

As mentioned in the introduction, the classification process is very subjective and therefore the output of this project is going to be equally subjective. A description has therefore been outlined so that conversational quality can be better anticipated, and adds some quantifiable properties to the classification process.

The classifier listened to multiple conversation recordings from the LDC data set and evaluated the quality of these conversations using a 7-point Likert scale ranging from terrible to excellent.

Data Formatting and Cleaning

Before the audio files could be computationally analysed, there was some editing and audio processing required in order to format and standardize the quality of the audio. This section describes the process involved.

Editing

It was noted that most participants had a similar way of starting and finishing a conversation that significantly altered their speech rate, tone of voice, and pitch in a way that did not resemble their usual speech behaviour. Therefore, introductions and farewells were omitted from the analysed conversations.

To maintain consistency across the experiment, conversations were cut to a length of 3 minutes. If there were any artefacts such as momentary distortion or clicks that might affect pitch or amplitude detection, this was also removed, provided that the artefact did not occur while someone was speaking.

Audio Production. As mentioned in Section 3.1, the LDC claimed the conversations were split into two channels. This condition was proven to work well in similar experiments for separate dyadic musical ensemble analysis and evaluation. However the split nature of the audio channels claim is not entirely true as there was a considerable amount of audio leaking from one channel into the other. The leaked audio was usually lower in volume and contained less lower frequencies. This allowed me to use a combination of side-chaining and gating/ducking techniques, where the audio of the affected signal is sent into a separate signal path where its high frequency content is removed. A ‘gate’ plugin listens to this channel and cuts off the signal if it does not hear anything below a certain frequency. This effectively turns down the volume (or turns ‘off’) the track whenever the main speaker of that channel is not speaking, as they are the only ones providing low-mid frequency content. Below are diagrams and visual representations for this setup. This process proved to be very time-consuming due to the fact that every voice is different and gate thresholds constantly had to be adjusted through trial and error.

Figure 2: Audio Processing Diagram

The audio channels were then bounced in wav files as split stereo channels, so that software analysis could be done on the voices separately. They were bounced at a 11025 KHz sample rate, as 8KHz causes issues with the software being used.

Feature Extraction

The first part of analysis is achieved using Praat(http://www.fon.hum.uva.nl/praat/)[2]. Pitch contour, silence/sound segmentation, word counts and jitter information was extracted from each speaker of the dyadic conversation individually. A Praat script was written to perform this process for batches of audio files automatically.Some of this script is an extension of a script created by [5] (original script at https://sites.google.com/site/speechrate/Home/praat-script-syllable-nuclei) which addresses the generation of silence/sound segmentation information. This information was then exported into csv files for further analysis in Matlab.

Pitch contours from Tony were compared with those of Praat. Tony seemed to have less mistakes, evidently restricting itself to realistic vocal behaviour. Conversely however, Tony’s pitch contours made poor choices in detecting when vocals were voiced and therefore forced fundamental frequency information out of non-pitched content, while Praat performed much better in this respect.

Using the MA Toolbox (https://code.soundsoftware.ac.uk/projects/tony http://www.ofai.at/elias.pampalk/ma) plugin for Matlab, amplitude information was extracted from the audio files for analysis.

Feature Analysis

With pitch contour, silence/sound segmentation, jitter and amplitude information extracted from the audio signals, further information relating to these attributes of each participant can be retrieved. With pitch contour, the mean and standard deviation values could be determined. With amplitude and back-channeling, standard deviation and mean values can be determined. With Silence/software and word counts, speech rate can be determined.

Devising a method for detecting backchannels was particularly challenging, as it required a significant amount of trial and error in order to get the parameters right. It was also noticed that threshold values had to be adjusted so that it matched the silence-sound report provided by Praat, leaving little room for tweaking. It was noted that most backchannels lasted between 200 and 1000 milliseconds, which interestingly contradicts the findings of [19].

Speech rate was calculated by summing the durations of the silence segments, subtracting this from the total duration of the conversation, and dividing the result by Praat’s word count.

All extractions were visualised in graphical representations as seen in Fig 3 in order to adjust settings to suit the behaviour of the speaker. All variables derived from each speakers voice was saved into a table for cross comparison. The variables collected at the end of analysis were: Pitch mean, pitch Std, Jitter, Sones mean, Sones Std, Silence duration, silence between speakers, overlapping between speakers, number of turns taken, number of backchannels used, total number of backchannels in conversation, word counts, and speech rate.

Figure 3: Visualisation of vocal analyses

Data Treatment

Once all of the data was collected for all relevant features, they were stored in a table with their associated conversation rating. Variables between participants, such as the frequency of the left speaker and frequency of the right speaker, were averaged to become one merged variable. For each rating (-3 to +3), scores for each variable were merged to provide an average for each type per rating.

4 RESULTS

The initial results did not explain a great deal about conversational behaviour. There was little correlation as you will see in the examples below. While the Matlab software was originally designed to work automatically without supervision, it was discovered that the tasks it was required to complete were too subjective, and judgments would vary greatly for each participant. Below is an example of inaccurate identifiers for backchannels and fundamental frequency.

Figure 4: Inaccurate representation of backchannels and frequency content

We can see by the blue circles in the graphs that backchannel points are inaccurately placed as well as the start of conversational chunks (red stars), while frequency ranges (green circles) are obscured by misidentifying octaves. In response to this, human supervision was included in the process so that judgments on correct identifications re- garding pitch, speech turns and back-channel recognition could be verified as acceptably accurate. Improvements in results verified the necessity for supervision. The corrections made to the previous analyses thanks to supervised analysis are presented below.

Figure 5: More accurate representation of backchannels and frequency content

Initially this project used a sample of 13 conversations. Due to the lack of clarity and correlations between the behaviour of variables and ratings, it was decided to extend the sample to include 16 conversations. It was also decided that the range of ratings should be reduced to four points, quantising the initial conversation evaluations to become the following, allowing for a more straightforward interpretation of the results:

0 Very poor

1 Poor

2 Good

3 Very Good

Upon this revision, correlations became more apparent which are presented below.

Mean pitch: r=0.929, p=0.07

Pitch Std: r=0.914, p=0.08

Jitter: r=-0.852, p=0.14

Mean Sones: r=0.942, p=0.05

Sones Std: r=0.98,p=0.01

Total silence per person: r=-0.946, p=0.05

Total Conversational Silence: r=-0.703, p=0.29

Total Conversational Overlap: r=0.598, p=0.41

Number of Turns: r=-0.647, p=0.35

Number of Backchannels per person: r=0.497, p=0.5

Total number of Backchannels: r=-0.483, p=0.51

Number of Words: r=0.966, p=0.03

Speech Rate: r=0.975, p=0.02

5 DISCUSSION

It is not surprising to see that frequency correlates somewhat positively with ratings. For many positively rated conversations, the speakers were often female so this will have been the main thing confounding variable influencing pitch. A solution is provided in the final section.

Sones had a strong correlation. This may be due to the fact that people speak more loudly when they are confident or passionate about something, although this assumption requires clarification. However, their standard deviation also correlates significantly which may be due to the fact that as [14] have verified, shimmer (which is shakiness in vocal amplitude) can be attributed to high arousal in emotions, which as described in the introduction could be strongly present in 'good' conversations.

Silence per person was a surprisingly high percentage. It was expected that this would go considerably lower than 60% for very good conversations but this was not the case, and can actually be explained by the research of [15] where they observe that silence in between words in speech make up a significant portion of speech.

Backchannel attributes did not correlate so well. Their values change direction once the rating increases from 'good' (rating:2). This could possibly be due to the fact that those who are very engaged in a conversation may be overly assuring their partner through backchannels, while those in bad conversations may be struggling might be offering backchannels in order to encourage the other speaker.

Words and speech rate correlated highly with ratings. This is most likely due to the fact that those who enjoy their conversation are more eager and passionate to get their point across quickly showing emotions of excitement or frustration. This would indeed determine a good conversation.

There are some other areas of correlation that are less strong and obvious, and there is not much point going into greater detail about these until the method has been expanded and improved upon as suggested in the following section

6 CONCLUSION AND FUTURE WORK

By evaluating conversations and observing the behaviour of extracted variables from the speakers, I have successfully been able to show that many of these variable behaviours correlate with the quality of a conversation. Using these correlating results as well as the feature extraction procedure, an algorithm can now predict conversational quality (of course based on the classifier's subjective understanding of what a good or bad conversation is).

It would not be surprising to see that a conversational quality prediction system might only work partially, with warped uncertainty due to the lack of training data. One obvious area for this of improvement for this project would be to increase the sample size dramatically. Due to the large amount of manual effort required to tweak settings and given the length of time available for this project, that was not feasible. Using a different data set (with truly separated sources and channels) would diminish the manual labour mentioned above significantly. [14] have suggested that attention should be drawn to raw signal, event density and brightness when detecting emotion, as these have been given less attention within the field, and could bare some fruitful insight. In future research, it would also be interesting to include attributes like these when analysing conversations.

The addition of multiple classifiers, cross-checking each others work during the evaluation period would also allow for a more generalised and consistent rating system, once the conditions and values were defined.

It is clear from the related work that psychological analysis would also be necessary to improve upon a conversational quality classifier. To include this would mean quantifying emotions directly. It may therefore also be worth analysing the behaviour of a single individual’s vocal cues under different conditions that would induce a wide variety of emotions. This will allow a baseline to be established for the nature and tendencies of a single voice, before subjecting it to a much higher-level analysis such as how the individual performs in conversation.

Analysing variables from individuals can also refer to their biological capabilities. Therefore a predictive system would also be in danger of favouring those that are more 'capable' than others of providing the optimum results (such as higher frequencies or amplitudes). For example in this data set, it was found that in general, female participants increased the chances of a conversation being rated positively. This is also worth taking into consideration. To start, it may be worth dividing dyadic conversation between the sexes. For now there are already plenty of avenues for further research and improvements.

7 ACKNOWLEDGEMENTS

The EPSRC and AHRC Centre for Doctiral Training in Media and Arts Technology (EP/L01632X/1) grant is funding the work presented in this article. Much guidance and reference material was provided by Queen Mary’s Professor Elaine Chew. Special thanks must go towards the Linguistics Data Consortium and Matthew Purver for providing assistance in obtaining the data sets necessary for this project. 

REFERENCES

[1] Anderson, C. M., and Rancer, A. S. The Relationship Between Argumentativeness, Verbal Aggressiveness, and Communication Satisfaction in Incarcerated Male Youth. The Prison Journal 87, 3 (Sept. 2007), 328–343.

[2] Boersma, P., and Weenink, D. Praat: Doing Phonetics by Computer, 2017.

[3] Carlson,R., Granström, B., and Nord, L. Experiments with emotive speech - acted utterances and synthesized replicas. In ICSLP (1992).

[4] Cosmides, L. Invariances in the Acoustic Expression of Emotion During Speech. Journal of Experimental Psychology- Human Perception and Performance 9, 6 (1983), 864–881.

[5] de Jong, N. H., and Wempe, T. Praat script to detect syllable nuclei and measure speech rate automatically. Behavior Research Methods 41, 2 (May 2009), 385–390.

[6] Dechert, H. W., and Raupach, M. Temporal Variables in Speech: Studies in Honour of Frieda Goldman-Eisler. De Gruyter, Inc., Berlin/Boston, GERMANY, 1980.

[7] Dittmann, A. T., and Llewellyn, L. G. Relationship between vocalizations and head nods as listener responses. Journal of Personality and Social Psychology 9, 1 (May 1968), 79–84.

[8] Guyer, J. J., Fabrigar, L. R., and Vaughan-Johnston, T. I. Speech Rate, Intonation, and Pitch: Investigating the Bias and Cue Effects of Vocal Confidence on Persuasion. Personality and Social Psychology Bulletin 45, 3 (Mar. 2019), 389–405. voices convey substantial information beyond the content of communications (Johnson, Ernde, Scherer, & Klinnert, 1986; Pell, Monetta, Paulmann, & Kotz, 2009).

[9] Hutchby, I., and Wooffitt, R. Conversation Analysis. Polity, Jan. 2008.

[10] Infante, D. A., and Rancer, A. S. A Conceptualization and Measure of Argumentativeness. Journal of Personality Assessment 46, 1 (Feb. 1982), 72–80.

[11]  Juslin, P. N., and Laukka, P. Communication of emotions in vocal expression and music performance: Different channels, same code? Psychological Bulletin 129, 5 (Sept. 2003), 770–814.

[12]  Kraus, M. W. Voice-only communication enhances empathic accuracy. American Psychologist 72, 7 (Oct. 2017), 644–654.

[13]  Laskowski, K., and Hjalmarsson, A. An information theoretic framework for automated discovery of prosodic cues to conversational structure. In 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) (Apr. 2015), pp. 5376–5380.

[14]  Lika, R. A., Seldon, H. L., and Kiong, L. C. Feature analysis of speech emotion data on arousal-valence dimension using adaptive neuro-fuzzy classifier. In 2014 International Conference on Industrial Automation, Information and Communications Technology (Aug. 2014), pp. 104–110.

[15]  Matarazzo, J.D., Hess, H.F., and Saslow,G. Frequency and Duration Characteristics of Speech and Silence Behavior During Interviews. Journal of Clinical Psychology 18, 4 (Oct. 1962), 416–426.

[16]  Matarazzo, J. D., Wiens, A. N., Saslow, G., Allen, B. V., and Weitman, M. Interviewer Mm-Hmm and interviewee speech durations. Psychotherapy: Theory, Research & Practice 1, 3 (1964), 109–114.

[17]  Meltzer, L., Morris, W. N., and Hayes, D. P. Interruption outcomes and vocal amplitude: Explorations in social psychophysics. Journal of Personality and Social Psychology 18, 3 (June 1971), 392–402.

[18]  Razak, A. A., Abidin, M. I. Z., and Komiya, R. Emotion pitch variation analysis in Malay and English voice samples. In 9th Asia-Pacific Conference on Communications (IEEE Cat. No.03EX732) (Sept. 2003), vol. 1, pp. 108–112 Vol.1.

[19]  Ward, N., and Tsukahara, W. Prosodic features which cue back- channel responses in English and Japanese. Journal of Pragmatics 32, 8 (July 2000), 1177–1207.