Hippocampal replay, or the off-line experience-dependent reactivation of hippocampal place cells [1-2] has been shown to contribute to learning and efficient decision making [3-5]. Assuming a reinforcement-learning model of the hippocampus, prioritized sweeping [6-7] has been observed to reproduce key properties of hippocampal replay [8], while also drastically increasing learning rates [9] via allocating resources to more uncertain regions of the internal model.
However, these models remain inherently reactive, as they cannot predict or anticipate changes in model uncertainty before the latter is observed. To alleviate this issue we propose a novel framework wherein classical reinforcement learning is augmented with supplementary dimensions encompassing internally generated epistemic rewards upon decreasing uncertainty in the internal world model. This allows the agent to predict delayed information gains, and actively decrease its own model uncertainty. Our solution predicts spatial preferences and replay patterns that are closer to experimentally observed hippocampal replay than previous models. Moreover, we use this framework to simulate novel situations in order to derive predictions for future experiments.
[1] M. Wilson and B. McNaughton, “Reactivation of hippocampal ensemble memories during sleep,” Science, vol. 265, no. 5172, pp. 676–679, Jul. 1994, doi: https://doi.org/10.1126/science.8036517.
[2] A. K. Lee and M. A. Wilson, “Memory of Sequential Experience in the Hippocampus during Slow Wave Sleep,”Neuron, vol. 36, no. 6, pp. 1183–1194, Dec. 2002, doi: https://doi.org/10.1016/s0896-6273(02)01096-6.
[3] K. Diba and G. Buzsáki, “Forward and reverse hippocampal place-cell sequences during ripples,” Nature Neuroscience, vol. 10, no. 10, pp. 1241–1242, Sep. 2007, doi: https://doi.org/10.1038/nn1961.
[4] G. Girardeau, K. Benchenane, S. I. Wiener, G. Buzsáki, and M. B. Zugaro, “Selective suppression of hippocampal ripples impairs spatial memory,” Nature Neuroscience, vol. 12, no. 10, pp. 1222–1223, Oct. 2009, doi: https://doi.org/10.1038/nn.2384.
[5] S. P. Jadhav, C. Kemere, P. W. German, and L. M. Frank, “Awake Hippocampal Sharp-Wave Ripples Support Spatial Memory,” Science, vol. 336, no. 6087, pp. 1454–1458, May 2012, doi: https://doi.org/10.1126/science.1217230.
[6] A. W. Moore and C. G. Atkeson, “Prioritized sweeping: Reinforcement learning with less data and less time,” Machine Learning, vol. 13, no. 1, pp. 103–130, Oct. 1993, doi: https://doi.org/10.1007/bf00993104.
[7] Jing Peng and R. J. Williams, “Efficient Learning and Planning Within the Dyna Framework,” Adaptive Behavior, vol. 1, no. 4, pp. 437–454, Mar. 1993, doi: https://doi.org/10.1177/105971239300100403.
[8] L. Aubin, M. Khamassi, and B. Girard, “Prioritized Sweeping Neural DynaQ with Multiple Predecessors, and Hippocampal Replays,” Lecture notes in computer science, pp. 16–27, Jan. 2018, doi: https://doi.org/10.1007/978-3-319-95972-6_4.
[9] R. D. Cazé, M. Khamassi, L. Aubin, and B. Girard, “Hippocampal replays under the scrutiny of reinforcement learning models,” Journal of Neurophysiology, vol. 120, no. 6, pp. 2877–2896, Dec. 2018, doi: https://doi.org/10.1152/jn.00145.2018.
| Type : | : | Poster |
| Thématiques | : | Computational |
| PDF version | : |  PDF version |
