Brain evidence and computational linguistics

My work on data-driven models of commonsense eventually led me to start looking into evidence from neuroscience, thanks above all to Heba Lakany, then a part of CSEE's BCI group. This line of research greatly expanded when I moved to the University of Trento's Center for Mind/Brain Sciences (CIMEC), where I started CLIC, a team of computer scientists working on language and commonsense knowledge in interaction with the neuroscientists and psychologists from the Centre. Over the years, we worked in four main areas: using EEG to study conceptual representations, using brain evidence to evaluate semantic space models, the representation of abstract concepts in the brain, using EEG for sentiment analysis, and on the clinical applications of this methodology.

Using EEG to study commonsense

The method most used to collect brain data about conceptual representations is functional Magnetic Resonance Imaging (fMRI), but collecting data with this evidence is very expensive, it could not be used for systematic studies of lexical representations. We have therefore explored the use of an alternative and much cheaper method, Electro Encephalography (EEG), much cheaper and already widely available in CS departments with BCI groups. We used this approach in many studies, first with Heba, then mainly in collaboration with Brian Murphy, and more recently with Yuqiao Gu. Other important collaborators include the members of Lorenzo Bruzzone's lab at Uni Trento, in particular Francesca Bovolo and Michele Dalponte.

Our first investigation in this area look at whether EEG evidence had sufficient discriminative power to replicate standard results in the MVPA literature--in particular, to differentiate between animals and tools. The results were positive (Murphy et al, 2011). We were also able to replicate our results using MEG (Murphy and Poesio, 2010).

Using brain evidence to evaluate semantic space models

Tom Mitchell and his lab pioneered a method to `map' conceptual representations extracted from corpora using distributional methods to activation patterns of concepts in the brain, using linear regression. Over the years we used several approaches to do this, including an approach to map distributional representations to data obtained using EEG (Murphy et al, 2009). More recently, we have mostly used representational similarity to do this (Kriegeskorte et al).

Brain evidence about abstract concepts

One area of the lexicon where neural evidence would be particularly welcome is the representation of abstract concepts such as concept, as this is the are of conceptual knowledge where there are the most theoretical disagreement between researchers working on commonsense knowledge and ontologies. Evidence in this respect is, however, still scarce. In work began with Brian and continued with Andrew Anderson and Yuan Tao, we have explored the representation of abstract concepts in the brain, testing in particular the predictions of Barsalou's theory (Anderson et al, 2014) and the implications of Pustejovsky's dot-objects theory of logic polysemy, in which words such as book may have both an abstract and a concrete interpretation (Tao et al, submitted).

Brain-driven sentiment analysis

Another aspect of lexical knowledge we have been studying using EEG and fMRI is the polarity of lexical items: negative, positive or neutral. Neuroscientists had already shown that polarity could be studied using fMRI in particular; again, we demonstrated that accuracy was beyond chance can be obtained with EEG, as well. In (Gu et al, 2014) we pioneered a new method to use polarities extracted this way for sentiment analysis, and showed that it is possible to obtain in this way better results than with standard sentiment lexica.

Cognitive diagnostics for semantic dementia

There are many types of semantic dementia, from Alzheimer to fronto-temporal dementia, each causing different types of damage to our cognitive abilities. In the ADAM project, we tested using the performance of the classifier with EEG data collected using the methods in (Murphy et al, 2011) to study brain aging and as an early diagnostic for Alzheimer (Gu et al, 2013).

Brain-evaluation of multimodal distributional analysis

In recent years, the focus of our research in this area has been the use of brain data to test multimodal distributional models (Anderson et al 2015, Anderson et al 2017).

Projects (in inverse chronological order)

  • PRESTO (2012-2016), a collaboration between Delta Informatica, Fondazione Bruno Kessler, and the University of Trento, is concerned with developing believable animated agents for training purposes. The role of CIMEC is to use experimental methods to study the impact of emotions on virtual agents.
  • ADAM (2011-2014), a collaboration between the Centre for Neurological Rehabilitation and CIMEC, is concerned with the development of non-invasive diagnostics for aging and semantic dementia.
  • Deep Relations (2011-2014), a collaboration with Expert Systems funded by the Provincia of Trento, was concerned with the development of innovative approaches to text mining. This project supported our pilot studies on brain-driven sentiment analysis.

Main publications