Behavior Inferencing

A critical part of AutoSense is enabling rich, real-time context inferencing using the sensors on the chestband and mobile phone. Real-time inferencing is needed to respond to errors and anomalies in the signals and display questionnaires to study participants in the field. The former is critical to collecting valid field data for scientific analysis, and the latter is critical for both collecting information from participants in the field and training or validating inferencing algorithms using participant responses.

We have several field studies planned that leverage AutoSense to study stress in the natural environment. Stress is a leading cause of mental and physical health problems. To reduce the occurrence of stress-related health problems, behavioral scientists need a better understanding of stress in the natural environment. Thus, AutoSense focuses specifically on inferring contexts associated with stress (e.g., stressors and coping behaviors). The following list represents the stress-related contexts we plan to infer in the field in real-time.

Intensity of Stress

We are working on real-time machine learning algorithms that infer a person's stress levels from their physiology. This work addresses two major challenges:

1. Previous work has shown that physiological stress responses vary from person to person. AutoSense will address this challenge by personalizing stress inferencing algorithms to each user. To personalize stress inferencing, we are running a lab study to collect participant physiological responses to stress. This personalized data will be used to train the stress inferencing algorithm. Two followup field studies with the same participant will be used to train and validate the personalized stress inferencing algorithm.
2. Previous work has also shown that physiological responses vary with respect to postures and physical activity. This variability makes it harder to infer a person's stress level since, for example, an increase in heart rate could mean the user is more stressed or it could mean the user stood up after lying down. AutoSense will address this challenge by automatically detecting these error states and either a) adjusting inferencing when the user transitions between error states, or b) shutting down real-time stress inferencing altogether. In the latter case, we shut down stress inferencing to preserve battery life on the mobile phone and motes.

This effort is being led by the Carnegie Mellon University team.

Emotion Classification

Smoking

To enable studying the relationship between stress and drug and alcohol use, we are developing an algorithm to infer when a user is smoking in real-time. Rather than use environment sensors, we are using a low-power respiration band that is part of AutoSense to search for a signature for smoking. The figure below compares the respiration signature of a person breathing quietly while working to a person smoking a cigarette. Clearly, smoking has a respiration signature that is different from other common respiration patterns, and we are working towards identifying this pattern in real-time to infer smoking. This research is being conducted by Somnath Mitra, Amin Ahsan Ali and Mahbubur Rahman.

Conversation

To enable studying how social aspects of everyday life affect stress, we are developing algorithms to infer conversation properties from the mobile phone microphone and the low-power respiration sensor. These properties include when the user is in conversation, when the user is speaking, when others are speaking, the number and genders of conversation participants, and the tenor of the conversation (serious, joyfyul, formal, informal, etc.). This research is being conducted by Amin Ahsan Ali and Mahbubur Rahman.

Several of these features will require the use the microphone on the AutoSense mobile phone. However, using the microphone has several drawbacks. First, the microphone is a relatively high-power sensor. It cannot be kept running continuously without sacrificing the battery life of the mobile phone. In addition, mobile phones are frequently kept in the pocket or purse. This means the microphone will often be covered such that it will be difficult to obtain a high quality audio signal for inferencing.

Thus, our current efforts focus on using the signal from the low-power respiration sensor to detect conversation. The microphone can then be used sparingly to infer specific properties of the conversation described previously (e.g. gender of conversation participants). The figure below shows that conversation has a different respiration signature than quiet breathing. Furthermore, this signature is also different from the smoking signature shown earlier. We are working towards identifying this pattern in real-time to infer smoking.

Interruptions

Interruptions are a significant source of stress in everyday life. To better understand the relationship between stress and interruptions, we are developing an algorithm to automatically detect interruptions. The human-computer interaction community has studied the harmful effects of interruptions extensively [1, 2]. Studies have identified when it is better to interrupt users [3, 4, 5] and tools that measure a person’s interruptibility with sensors [6, 7, 8, 9, 10]. In this work, we expand the body of knowledge on interruptions by measuring and predicting the physiological stress response from interruptions in the natural environment. In addition, we leverage modern sensing tools to capture the contexts in which such interruptions occurs, to better understand how stress varies from context to context.

As there are many types of interruptions, we focus specifically on interruptions from 1) ecological momentary assessments (EMA) and 2) conversation. The former will enable scientists to reduce the burden of EMA on study participants EMA, and the latter will enable development of stress/interruption-aware people and devices. People need better awareness of stress to help them manage everyday stress. Likewise, stress-aware devices will be be capable of predicting and responding to stressful events before the user can. For example, a stress-aware mobile phone will be capable of holding all calls from the boss - a stressor for some individuals - when the user is already quite stressed.

To detect interruptions from conversation, we leverage the conversation inferencing described previously. In addition, we hypothesize interruptions will coincide with a unique physiological "startle" response. Thus, our efforts focus on detecting the startle response and conversation from respiration signals, as well as merging these two events to enable high accuracy detection of everyday interruptions. This work is led by Andrew Raij in collaboration with Amin Ahsan Ali and Mahbubur Rahman.

[1] D. McFarlane. Comparison of four primary methods for coordinating the interruption

of people in human-computer interaction. Human-Computer Interaction,

17(1):63–139, 2002.

[2] D. McFarlane and K. Latorella. The scope and importance of human interruption

in human-computer interaction design. Human-Computer Interaction, 17(1):1–

61, 2002.

[3] B. P. Bailey and S. T. Iqbal. Understanding changes in mental workload during

execution of goal-directed tasks and its application for interruption management.

ACM Trans. Comput.-Hum. Interact., 14(4):1–28, 2008.

[4] M. Danninger, E. Robles, A. Sukumaran, and C. Nass. The Connector Service:

Representing Availability for Mobile Communication, chapter 19, pages 235–56.

Computers in the Human Interaction Loop. Springer London, 1 edition, 2009.

[5] S. Iqbal and B. Bailey. Investigating the effectiveness of mental workload as

a predictor of opportune moments for interruption. In Conference on Human

Factors in Computing Systems, pages 1489–1492. ACM New York, NY, USA,

2005.

[6] J. Fogarty, S. Hudson, C. Atkeson, D. Avrahami, J. Forlizzi, S. Kiesler, J. Lee,

and J. Yang. Predicting human interruptibility with sensors. ACM Transactions

on Computer-Human Interaction (TOCHI), 12(1):119–146, 2005.

[7] J. Fogarty, S. Hudson, and J. Lai. Examining the robustness of sensor-based statistical

models of human interruptibility. In Proceedings of the SIGCHI conference

on Human factors in computing systems, pages 207–214. ACM New York, NY,

USA, 2004.

[8] J. Ho and S. Intille. Using context-aware computing to reduce the perceived burden

of interruptions from mobile devices. In Proceedings of the SIGCHI conference

on Human factors in computing systems, pages 909–918. ACM New York,

NY, USA, 2005.

[9] E. Horvitz and J. Apacible. Learning and reasoning about interruption. In ICMI

’03: Proceedings of the 5th international conference on Multimodal interfaces,

pages 20–27, New York, NY, USA, 2003. ACM.

[10] A. Kapoor and E. Horvitz. Experience sampling for building predictive user models:

a comparative study. In CHI ’08: Proceeding of the twenty-sixth annual

SIGCHI conference on Human factors in computing systems, pages 657–666,

New York, NY, USA, 2008. ACM.

Physical Activity and Posture

High-accuracy detection of physical activity and posture are essentially solved problems, and thus we use previously developed algorithms to infer them [1, 2, 3]. We infer physical activity and posture because they change the physiological signals which our other inferencing algorithms depend on (e.g., stress). These changes introduce errors in other inferencing algorithms. Real-time detection of physical activity and posture will be used to activate and deactivate context inferencing based on the expected quality of data produced in various postures and activities.

[1] Maurer A et al ‐ Activity recognition and monitoring using multiple sensors on different body positions ‐ 2006.

[2] Bonomi, AG et al ‐ Detection of Type, Duration, and Intensity of Physical Activity Using an Accelerometer ‐ 2009

[3] Bao, L; Intille, S S ‐ Activity Recognition from User‐Annotated Acceleration Data ‐ 2004

Light Exposure

We infer light level because it is known to affect mood, circadian rhythms, and stress levels. While commodity mobile smartphones often have light sensors, it is difficult to reliably detect light if the phone is in the user's pocket or purse. Thus, we are exploring alternative approaches to augment the phone's light sensor in these situations.

Sound Level

Sound level is also known to affect mood and stress levels. Sound level is easily inferred from the microphone on the mobile phone.