My work on data-driven models of commonsense eventually led me to start looking into evidence from neuroscience, thanks above all to Heba Lakany, then a part of CSEE's BCI group. This line of research greatly expanded when I moved to the University of Trento's Center for Mind/Brain Sciences (CIMEC), where I started CLIC, a team of computer scientists working on language and commonsense knowledge in interaction with the neuroscientists and psychologists from the Centre. Over the years, we worked in four main areas: using EEG to study conceptual representations, using brain evidence to evaluate semantic space models, the representation of abstract concepts in the brain, using EEG for sentiment analysis, and on the clinical applications of this methodology.
The method most used to collect brain data about conceptual representations is functional Magnetic Resonance Imaging (fMRI), but collecting data with this evidence is very expensive, it could not be used for systematic studies of lexical representations. We have therefore explored the use of an alternative and much cheaper method, Electro Encephalography (EEG), much cheaper and already widely available in CS departments with BCI groups. We used this approach in many studies, first with Heba, then mainly in collaboration with Brian Murphy, and more recently with Yuqiao Gu. Other important collaborators include the members of Lorenzo Bruzzone's lab at Uni Trento, in particular Francesca Bovolo and Michele Dalponte.
Our first investigation in this area look at whether EEG evidence had sufficient discriminative power to replicate standard results in the MVPA literature--in particular, to differentiate between animals and tools. The results were positive (Murphy et al, 2011). We were also able to replicate our results using MEG (Murphy and Poesio, 2010).
Tom Mitchell and his lab pioneered a method to `map' conceptual representations extracted from corpora using distributional methods to activation patterns of concepts in the brain, using linear regression. Over the years we used several approaches to do this, including an approach to map distributional representations to data obtained using EEG (Murphy et al, 2009). More recently, we have mostly used representational similarity to do this (Kriegeskorte et al).
One area of the lexicon where neural evidence would be particularly welcome is the representation of abstract concepts such as concept, as this is the are of conceptual knowledge where there are the most theoretical disagreement between researchers working on commonsense knowledge and ontologies. Evidence in this respect is, however, still scarce. In work began with Brian and continued with Andrew Anderson and Yuan Tao, we have explored the representation of abstract concepts in the brain, testing in particular the predictions of Barsalou's theory (Anderson et al, 2014) and the implications of Pustejovsky's dot-objects theory of logic polysemy, in which words such as book may have both an abstract and a concrete interpretation (Tao et al, submitted).
Another aspect of lexical knowledge we have been studying using EEG and fMRI is the polarity of lexical items: negative, positive or neutral. Neuroscientists had already shown that polarity could be studied using fMRI in particular; again, we demonstrated that accuracy was beyond chance can be obtained with EEG, as well. In (Gu et al, 2014) we pioneered a new method to use polarities extracted this way for sentiment analysis, and showed that it is possible to obtain in this way better results than with standard sentiment lexica.
There are many types of semantic dementia, from Alzheimer to fronto-temporal dementia, each causing different types of damage to our cognitive abilities. In the ADAM project, we tested using the performance of the classifier with EEG data collected using the methods in (Murphy et al, 2011) to study brain aging and as an early diagnostic for Alzheimer (Gu et al, 2013).
In recent years, the focus of our research in this area has been the use of brain data to test multimodal distributional models (Anderson et al 2015, Anderson et al 2017).