Combining a coarse-and fine grained analysis to visualizing online meetings

This research project is an ongoing research activity in collaboration with the FlashMeeting group at the Knowledge Media Institute, Open University, UK.

Abstract

We describe the first steps towards the automation of video-conferencing data analysis for multi-party videoconferencing. The long-term goal is to use FlashMeeting and its embedded analysis tools for explaining ’what goes on’ in a requirements engineering analysis meeting at both coarse-grained and fine-grained levels.

Background

Meeting ‘minutes’ notoriously fail to capture the gestalt of what really ‘goes on’ during a meeting. Minutes can in principle capture the gist (depending on the skill of the editor/scribe), but are vulnerable to wide variation and even abuse by overzealous editors. Verbatim transcripts (if available) can be daunting in the amount of detail they yield, making it difficult to see the wood for the trees. We are interested in a ‘middle-tier’ phenomenological account of meetings that conveys key inflection points (such as ‘at 34:24 J interrupted Z with a clarification question’ or ‘at 15:10 R explained W to alleviate M’s confusion’). Such moments are noteworthy for online learning communities, and of great practical benefit for requirements engineering meetings, where misunderstandings can lead to software failures. For example for online learning communities, imagine that a student has to sit through hundreds (or even thousands) of video-meetings. Not all lectures are of real interest or in relationship to what a student is looking for. Hence we need to develop a 'caricature' that gives a summary of topics, and situations during the meetings.

We have already made some advances in the automatic analysis of meetings at a coarse-grained level (e.g. visualizing who dominated a meeting). In (Scott et al, 2007) the authors explored the ‘broadcast and chat dominance’ visualizations of a sample of different types of online meetings, noting that different polar area diagrams readily depicted different characteristic meeting ‘footprints’. The meetings analyzed were naturalistic events, held via FlashMeeting.

FlashMeeting Data Analysis- a macro analysis

FlashMeeting is a browser-embeddable Flash application that undertakes large-scale multi-party videoconferencing. It attains its scalability by enforcing half-duplex push-to-talk which allows only one person to broadcast video and audio at any one time. It also allows events to be automatically recorded. A public repository of those recorded events can be replayed by anyone online here, or have a look at the snapshots below:

Fig.1. FlashMeeting Memo. Replay your recorded online meetings.

These meetings are categorized according to theme tags (e.g. animeeting is the tag describing a family of meetings from a group of animation students). Refer to Fig.1, using FlashMeeting Memo, a user can click and replay on any part of a classic time-line/’music score’ visualization in which each horizontal bar represents one turn of a named user (Scott et al, 2007). In (Scott et al, 2007) we explored visualizations of different types of meetings using polar area diagrams to represent broadcast and chat dominance by subdividing the total time of the meeting into a pie chart, in which each area is proportional to the time that a particular person was speaking or text-chatting.

Fig.2. Broadcast dominance. Excerpted from (Scott et al, 2007, pg.6)

Fig.3. Chat dominance. Excerpted from (Scott et al, 2007, pg.6)

Fig.2 and Fig.3 shows a polar diagram of broadcast and chat dominance. At this coarse-grained level it is already possible to see different meeting types by their characteristic polar area diagram ‘footprints’. Nonetheless, we are still unable to fully explain what actually ‘went on’ in the meeting. The next section discusses our proposal to attack this problem by embedding a fine-grained analysis.

Fine-grained analysis

Firstly, we decided to categorize different situations occurring in different meetings into genres (e.g. play, sharing experience, problem solving). Within each genre, we focus on understanding what were the communication ‘characteristics’. For example, do project meetings exhibit any differences in problem solving when compared with, say, animation discussions? Hence, we retrieved 2 different types of meetings as a benchmark, peer-to-peer student meetings and research project meetings, to capture those characteristics explicitly by breaking them down into speech acts and topics. Here we used activity states to annotate the communications. Then we use the communication model together with emergent coding to organize the meetings into situations, and topic.

We show an excerpt of our results, consider Fig.4, row 1. ‘Speaker’ SC, at about ‘time’ 33:32, was broadcasting on ‘audio’ is marked up with ‘speech act’ in column 3. For example, inform-ref is informing in reference to ‘someone mentioning’ followed by describe what the person spoke about. The topic discussed was about ‘body pose’. In parallel to the chat, at ‘Time/Speaker’ 33:25 MH said that she lost feed. In the bottom row, SC at about 34:42 on ‘audio’ is talking about ‘good at freezing time’. SC is making an audio reply to a ‘side comment’ made by JW in text chat.

Fig. 4. Overview of what is happening in a meeting.

Now, the next step is to relate the analysis to ‘situations’ belonging to specific genres: play, sharing experience, discussion etc. And within these genres, what the topic of conversations was about. When we combined our fine-grained analysis to the coarse-grained analysis, we found discrepancies about the meeting. For example, although through the polar diagram it was revealed that speaker SC was dominant, we assumed he played the role as a mentor through out the meeting. At the coarse-grained analysis, speaker SC who was labeled as dominant, is actually dominant because at the start of the meeting, he was broadcasting alone while waiting for others to join the meeting. This was revealed from the fine-grained analysis. Moreover, the speaker left the meeting halfway and another speaker SR picked up the role as a 'group mentor and became 'dominant' until the end of the meeting.

Remaining challenges

We have already developed a fine-grained analysis framework that encompasses a communication model analysis within an emergent coding to extract situations and topics from the meetings. Our future direction is to explore the relationship between the coarse and the fine-grained analysis. Particularly, relating a participant’s projection of broadcast/chat dominance to the associative patterns between speech acts and topics. Will the relationship show an adaptation/merging of roles (i.e., from being a learner on body pose technique to a mentor on sharing past experiences on animation courses) taking place after some time through the social interaction? The next challenge would be to automate the analysis into the FlashMeeting replay. This would involve automatically transcribing the audio into text file, and then automating our manual fine-grained analysis that aligns the annotated chat with the transcribed audio file to arrive at the same results as we did manually.

Collaborators at Knowledge Media Institute (KMi), Open University, UK:

Prof Peter Scot, director of KMi

Dr. Eleftheria Tomadaki, research fellow

Dr. Kevin Quick, research fellow

Chris Valentine, project officer

Prof Marc Eisenstadt, previous chief scientist at KMi (the one who came up with the brilliant idea to combine micro and macro analysis!)