Second Interdisciplinary Symposium on Information Seeking, Curiosity and Attention (Neurocuriosity 2016)

Curiosity in humans and animals
#6

Matthias Gruber, Cardiff University, UK
The neurocognitive mechanisms of curiosity states on learning

Abstract: Little is known about the neural mechanisms by which curiosity affects learning. In a series of experiments, I will present evidence that show how curiosity states influence long-term memory. We found that participants showed improved memory for information that they were curious about, but also for incidental material learned during states of high curiosity. Results from an FMRI revealed increased activity in areas associated with the dopaminergic circuit during states of high curiosity. Importantly, individual variability in curiosity-driven memory benefits for incidental material was supported by anticipatory activity in the dopaminergic midbrain and hippocampus. In addition, evidence from an EEG and several behavioral experiments further characterize how curiosity enhances the drive to learn. In general, the findings highlight the importance of stimulating curiosity in order to create more effective learning experiences.

Back to the symposium’s table of contents

#7

Kou Murayama, University of Reading, UK
Curiosity as a complementary reward for extrinsic incentives

Abstract: The current presentation addresses a preliminary theory called “Curiosity as a complementary reward for extrinsic incentives”. The basic idea is simple — the theory posits that curiosity involves reward processes that are internally generated when extrinsic incentives are not available. Extrinsic incentives play an important role to shape our behavior, but extrinsic incentives are not always available, especially when someone is engaged in higher-order activities (e.g., reasoning, creativity). The theory argues that, in the evolutionary process, humans (and other higher organisms) acquired the ability to self-generate rewards to sustain their behavior when extrinsic incentives are not available and this is what we naively call, curiosity or interest. Based on this idea, I will present some preliminary evidence showing that (1) curiosity involves internal reward processing, (2) the effect of curiosity or interest accumulates over time, and (3) the rewarding process is elicited only when extrinsic incentives are not explicitly available.

Back to the symposium’s table of contents

#8

Jill O’Reilly, University of Oxford, UK
Control of entropy in internal models.

Abstract: Observers constantly build and update internal models of their environment, which are used to guide action. One challenge faced by these observers is to balance the need to construct stable models, based on a long history with data, against the need to be adaptable to possible changes in the environment. The latter is particularly relevant when the observer is actively exploring its environment. This talk will discuss situations in which there is a need to maintain flexibility in internal models (In Marr’s terms - Computational Theory), how this flexibility can be expressed mathematically (algorithm) and possible neural mechanisms (implementation) using data from human neuroimaging and behaviour in learning tasks.

Back to the symposium’s table of contents

#9

Karl Friston, UCL, UK
Active inference and artificial curiosity

Abstract: This talk offers a formal account of insight and learning in terms of active (Bayesian) inference. It deals with the dual problem of inferring states of the world and learning its statistical structure. In contrast to current trends in machine learning (e.g., deep learning), we focus on how agents learn from a small number of ambiguous outcomes to form insight. I will simulations of abstract rule-learning and approximate Bayesian inference to show that minimising (expected) free energy leads to active sampling of novel contingencies. This epistemic, curiosity-directed behaviour closes ‘explanatory gaps’ in knowledge about the causal structure of the world; thereby reducing ignorance, in addition to resolving uncertainty about states of the known world. We then move from inference to model selection or structure learning to show how abductive processes emerge when agents test plausible hypotheses about symmetries in their generative models of the world. The ensuing Bayesian model reduction evokes mechanisms associated with sleep and has all the hallmarks of ‘aha moments’.

Key words: active inference ∙ cognitive ∙ curiosity ∙ free energy ∙ epistemic value ∙ self-organization

Back to the symposium’s table of contents

#10

Gianluca Baldassarre, ISTC-CNR, Italy
Bio-inspired computational models of the development of attention skills: intrinsic motivations, goals, and learning

Abstract: I will first introduce a novel European Project supporting this research (``GOAL-Robots’’), and then present some computational models aiming to identify the cognitive architecture and mechanisms that might underlay the autonomous development of overt attention skills in children and robots. The models propose possible key functions, and underlying mechanisms, needed for the autonomous acquistion of attention skills: a fixed stimulus-based bottom-up attention component, a reinforcement learning top-down attention component, the coupling of eye/arm control, the self-generation of goals, intrinsic motivationsguiding autonomous learning and their coordination with extrinsic motivations. The modelswere developed using three sources of constraints; (a) behavioural functions/learning processes that, within embodied systems, seem broadly needed to produce the observable development of children; (b) data from developmental psychology experiments; © knowledge on the broad organisation of the brain system underlying attention control. Overall, the models represent tools for interpreting existing empirical experiments andsuggesting new ones.

Back to the symposium’s table of contents

#11

Peter Dayan, UCL, UK
Exploration bonuses: Bayes and Beyond

Abstract: A Bayesian view of curiosity is that it arises from exploration bonuses that depend on sophisticated prior distributions over the environment, including the relationships between multiple tasks and the length of time that they each can expect to be enjoyed or endured. Such priors and calculations lie most naturally in the purview of model-based reasoning. I will review some of our work along these lines, and draw links to simpler, less normative, approaches involving heuristics that might find model-free realization.

Slides: https://www.dropbox.com/s/8ju9ypw6d3hgyuw/dayan-neurocuriosity.pdf?dl=0

Back to the symposium’s table of contents

#12

Pierre-Yves Oudeyer, Inria, France
Diversity of forms and developmental functions of curiosity-driven learning

Abstract: Most studies about curiosity in children and adults focus on how attentional processes are driven by new or suprising external stimuli on a short time scale. However, theoretical developments using computational models have highlighted other very important dimensions of curiosity. Through discussing these theoretical approaches, I will argue that curiosity-driven exploration can take forms that go much beyond active sampling of external stimuli: I will explain that brain mechanisms for curiosity can also drive the active exploration of actions, of goals, of problem families, of problem spaces, of learning strategies, and of sources of information
such as when actively probing information from social peers. Then, I will also highlight that these processes should not only be studied within a short time scale, as computational models suggest they can have profound impact on long-term development. In particular, curiosity-driven learning can self-organize learning phases of increasing complexity, with spontaneous acquisition of skills that can be key in solving extrinsic problems where rewards are deceptive or rare.

Slides: https://www.dropbox.com/s/hmq95swxj8mibyk/neurocuriositylondonOudeyer16.pdf?dl=0

Back to the symposium’s table of contents

#13

Kenji Doya, OIST, Japan
Design, inference and evolution of reward functions for robots

Abstract: In applying reinforcement learning, the design of the reward function critically affects the speed of learning and the final preformance. The nature of reward functions in animals and humans is also an important question in biology and cognitive science. We are addressing the issue of reward design from both evolutionary robotics and statistical machine learning perspectives.
In living organisms, rewards are linked directly or inderectly with two defining features of life: self-preservation and reproduction. We developed a colony of robots that can survive by foraging battery packs and reproduce by copying their programs or parameters by infrared communication (Doya & Uchibe, 2005). By implementing a distributed evolutionary framework, we observed that robots acquire appropriate reward functions for foraging and mating, as well as the parameters for reinforcement learning and innate behaviors (Elfwing et al., 2011).
The inference of the reward function from observation of behaviors is formulated as inverse reinforcement learning, which is in genral ill-posed and intractable. Under the framework of linearly solvable MDP, we developed an efficient inverse reinforcement learning algorithm and demonstrated its performance with human and animal behavioral data (Uchibe & Doya, 2014). A recent extension allows its implementation with deep neural networks so that hand-coding of relevant features is not necessary (Uchibe, 2016).

Back to the symposium’s table of contents

#14

Wolfram Schultz, University of Cambrdige, UK
Risky Neurones

Abstract: Probability distributions of reward value offer a formal approach to the study of risk, which can be assessed as variance and skewness. Our monkeys’ risk attitudes are predicted from formal economic utility functions estimated from choices under variance risk. The animals’ choices are meaningful in following the gambles’ first, second and third order stochastic dominance. Single neurons in orbitofrontal cortex signal variance risk mostly separately from value; they signal also prediction errors in risk, which might be useful for the updating of risk information. The terms ‘risk avoidance’ and ‘risk taking’ suggest that risk affects the subjective value of rewards. Correspondingly, the dopamine reward prediction error response reflects changes in the subjective reward value derived from different levels of risk. More formally, the signal codes economic utility and satisfies first- and second-order stochastic dominance.

Back to the symposium’s table of contents

#15

Andrew Bell, University of Oxford, UK
How Face Neurons contribute to Perceptual Decision Making

Abstract: Existing in a social environment requires that we have some ability to assess the mental states of others. We do this by interpreting facial expressions, body language, and other social cues and using this information to guide our own decisions and actions. Using a combination of neuroimaging and electrophysiological techniques, my lab seeks to understand the underlying neural pathways and mechanisms involved in this behaviour. In this talk, I will discuss what we currently know about face neurons, including where they are found in the brain and what makes them so “special”. In particular, I will describe data from our lab that seeks to understand how they might contribute to choice behaviour in a manner consistent with a popular decision neuroscience framework known as predictive coding.

Back to the symposium’s table of contents

#16

Laurence Hunt, UCL, UK
Reward-guided information search and choice in prefrontal cortex

Abstract: Many studies emphasise the importance of prefrontal cortex (PFC) in reward-guided decision-making. These studies often use 2-alternative forced choice, ignoring the inherently sequential nature of most real-world decisions. Here we study behavioural and neural data from a choice task in which subjects must decide which information and how much information to gather, before deciding which option to choose. In humans, we collected behavioural data from 30,000 subjects via smartphone, revealing systematic biases in how we sample information for rewards. These were formalized by comparing subject behavior to a dynamic programming model of optimal information gathering. In non-human primates, we recorded activity from 724 single neurons in several subregions of prefrontal cortex (ACC, OFC, DLPFC). Distinct PFC subregions reflected the decision over whether the currently attended option should be selected, the spatial deployment of attention, or whether newly sampled information confirmed one’s current beliefs about the decision at hand.

Back to the symposium’s table of contents

#17

Jacqueline Gottlieb, Columbia University, US
Day 2 General Discussion: Electrophysiology

Back to the symposium’s table of contents

#18

Derek Bell, Learnus, UK
So what do I do in my lessons next week?

Abstract: "So what do I do in my lessons next week?” is a frequent question from teachers when introduced to findings from the cognitive sciences including educational neuroscience. The challenge of bridging the gap between research and practice is not unique to education but, as the drive towards evidence-based approaches increases, it is a major challenge for teachers and researchers alike. This presentation will endeavour to highlight some of the issues that need to be addressed translating research findings into actual classroom practice. Does the evidence indicate that there needs to be major changes in teachers’ practice or are more subtle adjustments required? Are there examples of current practice that build on the evidence effectively? What needs to happen in schools in order to take advantage of improvements in our understanding of learning? What channels are there available in order to introduce these changes into classroom practice? How can teachers work more effectively with researchers in the cognitive sciences to address the issues? In short how can we give an effective answer to the question “So what do I do in my lessons next week?”

Back to the symposium’s table of contents

#19

Katarina Begus, Central European University, Hungary
Active learners: Behavioural and neural mechanisms of selective social learning in infancy

Abstract: Even in their first year of life, infants appear to allocate their attention and guide their exploration in a way that ensures maximal information gain. While young children’s active role in gathering information and generating evidence has been an enduring theme of investigation in the domain of children’s play and exploration, it has been largely neglected in the field of social learning. Given that learning from people is one of the most prominent ways in which infants acquire knowledge in everyday life, the research presented in this talk aimed at investigating infants’ active involvement in the process of social transmission of knowledge. A series of studies, using both behavioural and neuroimaging methods, demonstrated that even very young infants are active social learners, with means of soliciting information from social partners and with mechanisms of selective engagement and learning based on their interests and reliability of information sources.

Slides: https://www.dropbox.com/s/2zmfe7dbg32pktx/Begus-neurocuriosity.pdf?dl=0

Back to the symposium’s table of contents

#20

Louise Goupil, ENS Paris, France
Developing a reflective mind: uncertainty monitoring, decision confidence and error detection in human infants

Abstract: Humans adapt their behavior not only by observing the consequences of their actions, but also by internally monitoring their own cognitive states. Because these metacognitive abilities are typically assessed through explicit self-reports, they were traditionally denied to young children, who can hardly verbalise their own cognitive states. Here, by relying on non-verbal indices, we show that human infants actually possess rudimentary forms of metacognition. In a first study, we show that 20-month-olds are able to ask for help non-verbally in order to avoid making errors. In a second study, we found that a well-established electrophysiological signature of error monitoring in adults is similarly elicited when 12-month-olds make a mistake. Hence,although explicit forms of metacognition mature later during childhood, infants can already monitor their own cognitive states, and use these metacognitive evaluations to regulate subsequent behavior.

Back to the symposium’s table of contents

#21

Lisa Feigenson, Johns Hopkins University, US
How Core Knowledge Drives Learning

Abstract: Across species and across human development, core knowledge empowers expectations about the physical and social world. Many non-human creatures—as well as very young human infants—exhibit knowledge about how objects should behave, how quantities should transform, and how social agents should act. This core knowledge has often been thought of as an alternative to learning. In contrast to this “static knowledge” view, I will describe recent findings from infants and children that show that core knowledge shapes the acquisition of new information. Across a range of learning content and tasks, babies and children show enhanced learning for entities that violated their basic expectations, and test hypotheses for these violations. These findings suggest that foundational knowledge shapes new learning.

Back to the symposium’s table of contents

#22

[Daphne Bavelier, University of Geneva, Switzerland and University of Rochester, US] (https://cms.unige.ch/fapse/people/bavelier/)
Learning to Learn: Lessons from Action Video Game Play

Abstract: A vexing issue in the field of learning is that, while we understand how to promote superior performance through practice, the resulting behavioral enhancement rarely extends beyond the practiced task. Such learning specificity is a major limitation for effective interventions, whether educational or clinical ones. Here we will consider first how learning and generalization may be enhanced, through a mechanism we term ‘learning to learn’ (L2L). We then ask what may be the determinants of ‘learning to learn’ – differentiating between adjusting parameters as learning of a specific task proceeds from extracting the structure across tasks to facilitate learning and generalization.

Back to the symposium’s table of contents

#23

Alison Gopnik, UC Berkeley, US
When (and why) children are more open-minded than adults: Childhood as simulated annealing

Abstract: I will present several studies showing that preschoolers can learn abstract higher-order principles from data. In each case, younger learners were actually better at inferring unusual or unlikely principles than older learners. I relate this to computational ideas about search and sampling, to evolutionary ideas about human life history, and to neuroscience findings about the negative effects of frontal control on wide exploration, and the advantages of earlier neural architectures for wide-ranging learning. Our hypothesis is that childhood is evolution’s way of performing simulated annealing. Our distinctively long human childhood allows a period of broad exploratory “high-temperature” hypothesis search.

Back to the symposium’s table of contents

#24

Teodora Gliga, Birkbeck, University of London, UK
Day 3 General Discussion: Developmental Psychology

Back to the symposium’s table of contents

#25

Day 2 Discussion: Computational theories of curiosity and active exploration in animals and machines

Slides: https://www.dropbox.com/s/ps41i5hgyyz88wk/discussionNeurocuriosityOudeyer16.pdf?dl=0