Distracted driving is a leading cause of accidents worldwide. The tasks of distraction detection and recognition have been traditionally addressed as computer vision problems. However, distracted behaviors are not always expressed in a visually observable way. In this project, we introduce a novel multimodal dataset of distracted driver behaviors, consisting of data collected using twelve information channels coming from visual, acoustic, near-infrared, thermal, physiological, and linguistic modalities. The data were collected from 45 subjects while being exposed to four different distractions (three cognitive and one physical). Data collection experiments took place during different times of the day to account for a greater drowsiness variability in our participants. Through the analysis of the data, we explore a variety of spatio-temporal machine learning techniques, ranging from traditional machine learning pipelines to more novel, state-of-the-art deep learning architectures. We identify modality-specific and multi-modal supervised and unsupervised features and we evaluate their ability to characterize the two different conditions within and across subjects. This project aims to explore how drowsy and distracted states overlap in drivers and how they can affect driver performance and safety. Our goal is to design personalized and adjustable AI models that learn through interaction and fit each individual's needs in order to increase road safety and offer just-in-time interventions. This project is in collaboration with the Toyota Research Institute.