Brain evidence and computational linguistics
My work on data-driven models of commonsense eventually led me to start looking into evidence from neuroscience, thanks above all to Heba Lakany, then a part of CSEE's BCI group. This line of research greatly expanded when I moved to the University of Trento's Center for Mind/Brain Sciences (CIMEC), where I started CLIC, a team of computer scientists working on language and commonsense knowledge in interaction with the neuroscientists and psychologists from the Centre. Over the years, we worked in four main areas: using EEG to study conceptual representations, using brain evidence to evaluate semantic space models, the representation of abstract concepts in the brain, using EEG for sentiment analysis, and on the clinical applications of this methodology.
Using EEG to study commonsense
The method most used to collect brain data about conceptual representations is functional Magnetic Resonance Imaging (fMRI), but collecting data with this evidence is very expensive, it could not be used for systematic studies of lexical representations. We have therefore explored the use of an alternative and much cheaper method, Electro Encephalography (EEG), much cheaper and already widely available in CS departments with BCI groups. We used this approach in many studies, first with Heba, then mainly in collaboration with Brian Murphy, and more recently with Yuqiao Gu. Other important collaborators include the members of Lorenzo Bruzzone's lab at Uni Trento, in particular Francesca Bovolo and Michele Dalponte.
Our first investigation in this area look at whether EEG evidence had sufficient discriminative power to replicate standard results in the MVPA literature--in particular, to differentiate between animals and tools. The results were positive (Murphy et al, 2011). We were also able to replicate our results using MEG (Murphy and Poesio, 2010).
Using brain evidence to evaluate semantic space models
Tom Mitchell and his lab pioneered a method to `map' conceptual representations extracted from corpora using distributional methods to activation patterns of concepts in the brain, using linear regression. Over the years we used several approaches to do this, including an approach to map distributional representations to data obtained using EEG (Murphy et al, 2009). More recently, we have mostly used representational similarity to do this (Kriegeskorte et al).
Brain evidence about abstract concepts
One area of the lexicon where neural evidence would be particularly welcome is the representation of abstract concepts such as concept, as this is the are of conceptual knowledge where there are the most theoretical disagreement between researchers working on commonsense knowledge and ontologies. Evidence in this respect is, however, still scarce. In work began with Brian and continued with Andrew Anderson and Yuan Tao, we have explored the representation of abstract concepts in the brain, testing in particular the predictions of Barsalou's theory (Anderson et al, 2014) and the implications of Pustejovsky's dot-objects theory of logic polysemy, in which words such as book may have both an abstract and a concrete interpretation (Tao et al, submitted).
Brain-driven sentiment analysis
Another aspect of lexical knowledge we have been studying using EEG and fMRI is the polarity of lexical items: negative, positive or neutral. Neuroscientists had already shown that polarity could be studied using fMRI in particular; again, we demonstrated that accuracy was beyond chance can be obtained with EEG, as well. In (Gu et al, 2014) we pioneered a new method to use polarities extracted this way for sentiment analysis, and showed that it is possible to obtain in this way better results than with standard sentiment lexica.
Cognitive diagnostics for semantic dementia
There are many types of semantic dementia, from Alzheimer to fronto-temporal dementia, each causing different types of damage to our cognitive abilities. In the ADAM project, we tested using the performance of the classifier with EEG data collected using the methods in (Murphy et al, 2011) to study brain aging and as an early diagnostic for Alzheimer (Gu et al, 2013).
Brain-evaluation of multimodal distributional analysis
In recent years, the focus of our research in this area has been the use of brain data to test multimodal distributional models (Anderson et al 2015, Anderson et al 2017).
Projects (in inverse chronological order)
- PRESTO (2012-2016), a collaboration between Delta Informatica, Fondazione Bruno Kessler, and the University of Trento, is concerned with developing believable animated agents for training purposes. The role of CIMEC is to use experimental methods to study the impact of emotions on virtual agents.
- ADAM (2011-2014), a collaboration between the Centre for Neurological Rehabilitation and CIMEC, is concerned with the development of non-invasive diagnostics for aging and semantic dementia.
- Deep Relations (2011-2014), a collaboration with Expert Systems funded by the Provincia of Trento, was concerned with the development of innovative approaches to text mining. This project supported our pilot studies on brain-driven sentiment analysis.
- Anderson, Andrew J, Douwe Kiela, Stephen Clark and Massimo Poesio, 2017. Visually grounded and textual semantic models differentially decode brain activity associated with concrete and abstract nouns. Transactions of the ACL.
- Anderson, Andrew J, Elia Bruni, Alessandro Lopopolo, Massimo Poesio and Marco Baroni, 2015. Reading visually embodied meaning from the brain: Visually grounded computational models decode visual-object mental imagery induced by written text. Neuroimage.
- Andrew Anderson, Brian Murphy, and Massimo Poesio, 2014. Discriminating taxonomic categories and domains in mental simulations of concepts of varying concreteness. Journal of Cognitive Neuroscience, 26(3), 658-681.. (pdf)
- Yuqiao Gu, Fabio Celli, Josef Steinberger, Andrew Anderson, Massimo Poesio, Carlo Strapparava and Brian Murphy, 2014. Using brain data for sentiment analysis. Journal of Linguistics and Computational Linguistics, 29(1), 79-94. (pdf)
- Hiroyuki Akama, Brian Murphy, Miao Mei Lei and Massimo Poesio, 2014. Cross-participant modelling based on joint or disjoint feature selection: an fMRI conceptual decoding study. Applied Informatics. 1(1). (pdf)
- Yuqiao Gu, Giulia Cazzolli, Brian Murphy, Gabriele Miceli, and Massimo Poesio, 2013. EEG study of the neural representation and classification of semantic categories of animals vs tools in young and elderly participants. BMC Neuroscience 29(1), 79-94.
- H. Akama, Brian Murphy, L. Na, Y. Shimizu and Massimo Poesio, 2012. Decoding semantics across fMRI sessions with different stimulus modalities: a practical MVPA study. Frontiers in Neuroinformatics, 6(24).
- Brian Murphy, Massimo Poesio, Francesca Bovolo, Lorenzo Bruzzone, Michele Dalponte, and Heba Lakany, 2011. EEG decoding of semantic category reveals distributed representations for single concepts. Brain and Language, 117(1), 12-22. (pdf)
- Anderson, A., B. Murphy and M. Poesio, 2012. Differences in fMRI representation of concrete and abstract concepts. In Proc. of the Neural Computation and Psychology Workshop (NCPW13), San Sebastian.
- Uryupina, O. and M. Poesio, 2012. Domain-specific vs. Uniform Modeling for Coreference Resolution. In Proc. of LREC, Istanbul.
- Brian Murphy and Massimo Poesio, 2010. Detecting Semantic Category in Simultaneous EEG/MEG Recordings. In Proceedings of the NAACL-HLT Workshop on Computational Neurolinguistics. (pdf)
- B. Murphy, M. Baroni and M. Poesio, 2009. EEG Responds to Conceptual Stimuli and Corpus Semantics. In Proc. of EMNLP, Singapore, July. (pdf.)
- Brian Murphy, Michele Dalponte, Massimo Poesio, and Lorenzo Bruzzone, 2008. Distinguishing Concept Categories on the Basis of Single-Participant electrophysiological activity. Proc. of The Annual Meeting of the Cognitive Science Society, Washington, July. ( pdf)