A multidimensional distributional map of future reward in dopamine neurons

• Published in: Nature (London) Vol. 642; no. 8068; pp. 691 - 699
• Main Authors: Sousa, Margarida, Bujalski, Pawel, Cruz, Bruno F., Louie, Kenway, McNamee, Daniel C., Paton, Joseph J.
• Format: Journal Article
• Language: English
• Published: London Nature Publishing Group UK 19.06.2025; Nature Publishing Group
• Subjects: 13/109; 631/378/116/2396; 631/378/1788; 64/60; Algorithms; Animals; Anticipation, Psychological - physiology; Coding theory; Cues; Dopamine; Dopaminergic Neurons - physiology; Female; Foraging behavior; Humanities and Social Sciences; Learning; Machine learning; Male; Mesencephalon; Mesencephalon - cytology; Mesencephalon - physiology; Mice; Mice, Inbred C57BL; Models, Neurological; multidisciplinary; Neurons; Optogenetics; Predictions; Probability; Probability distribution; Reinforcement; Reinforcement, Psychology; Reward; Science; Science (multidisciplinary); Statistical analysis; Time Factors
• ISSN: 0028-0836; 1476-4687; 1476-4687
• DOI: 10.1038/s41586-025-09089-6

Midbrain dopamine neurons (DANs) signal reward-prediction errors that teach recipient circuits about expected rewards 1 . However, DANs are thought to provide a substrate for temporal difference (TD) reinforcement learning (RL), an algorithm that learns the mean of temporally discounted expected future rewards, discarding useful information about experienced distributions of reward amounts and delays 2 . Here we present time–magnitude RL (TMRL), a multidimensional variant of distributional RL that learns the joint distribution of future rewards over time and magnitude. We also uncover signatures of TMRL-like computations in the activity of optogenetically identified DANs in mice during behaviour. Specifically, we show that there is significant diversity in both temporal discounting and tuning for the reward magnitude across DANs. These features allow the computation of a two-dimensional, probabilistic map of future rewards from just 450 ms of the DAN population response to a reward-predictive cue. Furthermore, reward-time predictions derived from this code correlate with anticipatory behaviour, suggesting that similar information is used to guide decisions about when to act. Finally, by simulating behaviour in a foraging environment, we highlight the benefits of a joint probability distribution of reward over time and magnitude in the face of dynamic reward landscapes and internal states. These findings show that rich probabilistic reward information is learnt and communicated to DANs, and suggest a simple, local-in-time extension of TD algorithms that explains how such information might be acquired and computed. An algorithm called time–magnitude reinforcement learning (TMRL) extends distributional reinforcement learning to take account of reward time and magnitude, and behavioural and neurophysiological experiments in mice suggest that midbrain dopamine neurons use TMRL-like computations.