puc-header

Prefrontal Solution to the Bias-Variance Tradeoff During Reinforcement Learning

61 Pages Posted: 24 Mar 2021 Publication Status: Review Complete

See all articles by Dongjae Kim

Dongjae Kim

Korea Advanced Institute of Science and Technology (KAIST) - Department of Bio and Brain Engineering

Jaeseung Jeong

Korea Advanced Institute of Science and Technology (KAIST) - Department of Bio and Brain Engineering

Sang Wan Lee

Korea Advanced Institute of Science and Technology (KAIST) - Department of Bio and Brain Engineering

More...

Abstract

The goal of learning is to maximize future rewards by minimizing prediction errors. Evidence have shown that the brain achieves this by combining model-based and model-free learning. However, the prediction error minimization is challenged by a bias-variance tradeoff, which imposes constraints on each strategy’s performance. We provide new theoretical insight into how this tradeoff can be resolved through the adaptive control of model-based and model-free learning. The theory predicts the baseline correction for prediction error reduces the lower bound of the bias–variance error by factoring out irreducible noise. Using a Markov decision task with context changes, we showed behavioral evidence of adaptive control. Model-based behavioral analyses show that the prediction error baseline signals context changes to improve adaptability. Critically, the neural results support this view, demonstrating multiplexed representations of prediction error baseline within the ventrolateral and ventromedial prefrontal cortex, key brain regions known to guide model-based and model-free learning.

Suggested Citation

Kim, Dongjae and Jeong, Jaeseung and Lee, Sang Wan, Prefrontal Solution to the Bias-Variance Tradeoff During Reinforcement Learning. Available at SSRN: https://ssrn.com/abstract=3811830 or http://dx.doi.org/10.2139/ssrn.3811830
This version of the paper has not been formally peer reviewed.

Dongjae Kim

Korea Advanced Institute of Science and Technology (KAIST) - Department of Bio and Brain Engineering

Korea, Republic of (South Korea)

Jaeseung Jeong

Korea Advanced Institute of Science and Technology (KAIST) - Department of Bio and Brain Engineering

Korea, Republic of (South Korea)

Sang Wan Lee (Contact Author)

Korea Advanced Institute of Science and Technology (KAIST) - Department of Bio and Brain Engineering ( email )

Korea, Republic of (South Korea)

Click here to go to Cell.com

Paper statistics

Abstract Views
46
Downloads
1
PlumX Metrics