Post date: Oct 21, 2012 12:28:26 PM
How does the brain select among actions, goals and plans?
Neuroscience and cognitive robotics studies have long focused on the neural underpinnings and computational models of the acquisition and selection of actions. Still, most studies have focused on what psychologists call "habitual" actions (e.g., when I am in front of my house-door, I almost invariantly search the keys; often, also when the door is open!) and associated neuro-computatal mechanisms (famously, linking dopamine function to the temporal-difference learning methods of reinforcement learning, Schultz et al., 1995).
Recently there is interest on the mechanisms underlying 'goal-directed' actions and choices. From a theoretical perspective, they have been associated to model-based methods of reinforcement learning (in contrast to model-free methods that better characterize 'habitual' actions, Daw et al., 2005). They are more flexible, but also more cognitively demanding, than habitual mechanisms (e.g., in front of my house, I can reason that because it is open I don't need the keys, but this requires some effort). Computationally, they can be characterized as using two mechanisms: one for generating predictions of the outcome of possible actions (say, what happens if I do this or that) and another for evaluating and comparing these outcomes, and selecting among them (say, it is better to do this than that). The neural underpinnings of goal-directed mechanisms of action and choice, long studied in rodents (Balleine and Dickinson, 1998) now begin to be assessed in humans as well (Simon and Daw, 2011). In this area, there is a long-lasting collaboration of empirical and computational modeling studies.
We have recently proposed the alternative idea that habitual and goal-directed systems are combined in a cooperative controller called the "mixed instrumental controller", where --when needed-- (prior) value information provided by the habitual system can be combined with additional information sampled from an internal model of the task following a Bayesian integration scheme (Pezzulo et al 2013). This idea is supported by recent findings that activity in human ventral striatum is better predicted by a combination of model-based and model-free values (Daw et al 2011).
This is a Special Issue on The principles of goal-directed decision-making: from neural mechanisms to computation and robotics I co-edited for the Phil. Trans. R. Soc. B. journal in 2014.
We are investigating several facets of goal-directed actions and choices within the EU-funded project GOAL-Leaders: Goal-directed, Adaptive Builder Robots (Goal-Leaders). In one stream of research, we are building neural models of how hippocampus and ventral striatum interact to implement goal-directed choices in navigation domains (implementing the two aforementioned mechanisms: predictors and evaluators, Chersi and Pezzulo, 2012). This activity is aimed at modeling so-called 'forward sweeps' in the rat hippocampus (Johnson and Redish, 2007), which is a plausible neuronal substrate of predictions and mental simulations for navigation. Furthermore, this activity sees the ventral striatum as a possible neuronal substrate for outcome evaluation and selection (Pennartz et al., 2011).
See a video here: http://videolectures.net/cogsys2012_pezzulo_builder/
In another stream of research, we are realizing a series of probabilistic (Bayesian) computational models of goal-directed choice (see Solway and Botvinick, 2012 for a related model). With these models, we study the mechanisms of prospection, future-directed choice, and the synergies between habitual and goal-directed mechanisms of choice (Pezzulo and Rigoli, 2011). Indeed, it is well known that goal-directed, habitual and other mechanisms (e.g., Pavlovian responses) could compete or work in synergy, depending on the situation. We are study this issue empirical as well (Rigoli et al., 2012).
Finally, we are studying how goal hierarchies (plausibly linked to prefrontal cortex functioning, Koechlin and Summerfield, 2007) make goal-directed mechanisms of choice so flexible and context-dependent and implement forms of cognitive control, casting these problems within the frameworks of "predictive coding" and "active inference" (Pezzulo, 2012; Pezzulo et al., 2018).
Selected Pubs:
Pezzulo G., Rigoli, F. Friston, K. (2018) Hierarchical Active Inference: a Theory of Motivated Control. Trends in Cognitive Sciences 22(4), 294-306 [link]
Pezzulo, G., Rigoli, F., Friston, K. (2015) Active Inference, homeostatic regulation and adaptive behavioural control. Progress in Neurobiology 134, 17-35 [link][pdf]
Pezzulo, G. (2012). An active inference view of cognitive control. Frontiers in Theoretical and Philosophical Psychology. [link]
Pezzulo, G. and Rigoli, F. (2011). The value of foresight: how prospection affects decision-making. Front. Neurosci., 5(79). [link]
Rigoli, F., Pavone, E. F., and Pezzulo, G. (2012). Aversive pavlovian responses affect human instrumental motor performance. Front. Neurosci., 6:134. [link]
Pezzulo, G., Rigoli, F., Chersi F. (2013) The Mixed Instrumental Controller: Using Value of Information to Combine Habitual Choice and Mental Simulation. Frontiers in Psychology 4:92. doi: 10.3389/fpsyg.2013.00092 [link]
Chersi, F. and Pezzulo, G. (2012). Using hippocampal-striatal loops for spatial navigation and goal-directed decision-making. Cognitive Processing, 13(1):125–129. [WEB]
Other Pubs:
Balleine, B. W. & Dickinson, A. (1998) Goal-directed instrumental action: contingency and incentive learning and their cortical substrates. Neuropharmacology, 37, 407-419
Daw, N. D.; Niv, Y. & Dayan, P. (2005) Uncertainty-based competition between prefrontal and dorsolateral striatal systems for behavioral control Nature Neuroscience, 8, 1704-1711
Daw, N. D.; Gershman, S. J.; Seymour, B.; Dayan, P. & Dolan, R. J. (2011) Model-based influences on humans' choices and striatal prediction errors. Neuron, 69, 1204-1215
Johnson, A. & Redish, A. D. (2007) Neural ensembles in CA3 transiently encode paths forward of the animal at a decision point. J Neurosci 27, 12176-12189
Koechlin, E. & Summerfield, C. (2007) An information theoretical approach to prefrontal executive function. Trends Cogn Sci 11, 229-235
Pennartz, C. M. A.; Ito, R.; Verschure, P. F. M. J.; Battaglia, F. P. & Robbins, T. W. (2011) The hippocampal-striatal axis in learning, prediction and goal-directed behavior. Trends Neurosci, 34, 548-559
Schultz, W.; Dayan, P. & Montague, P. A neural substrate of prediction and reward Science, 1997, 275, 1593-1599
Simon, D. A. & Daw, N. D. (2011) Neural correlates of forward planning in a spatial decision task in humans. J Neurosci, 31, 5526-5539
Solway, A. & Botvinick, M. M. (2012) Goal-directed decision making as probabilistic inference: A computational framework and potential neural correlates. Psychol Rev,119, 120-154