Temporal Abstraction in Reinforcement Learning with the Successor Representation
Authors: Marlos C. Machado, Andre Barreto, Doina Precup, Michael Bowling; Volume 24, Issue 80, Pages 1-69, 2023.
Abstract
Intelligence often involves reasoning at multiple levels of temporal abstraction. In the field of reinforcement learning, this is achieved through the use of options, which are temporally extended courses of actions. Options allow agents to make predictions and operate at different levels of abstraction within an environment. However, existing approaches based on the options framework typically assume that a predefined set of options is known. When this is not the case, determining which options to consider becomes a challenge. In this paper, we propose that the successor representation, which encodes states based on the pattern of state visitation, can be utilized as a natural substrate for the discovery and utilization of temporal abstractions. We present a comprehensive overview of recent results that demonstrate how the successor representation can be employed to discover options that facilitate temporally extended exploration and planning. We frame these results within a general framework for option discovery, where the agent’s representation is used to identify useful options that further enhance its representation. This creates a continuous cycle where both the representation and the options are constantly refined based on each other. In addition to option discovery, we discuss how the successor representation enables the augmentation of a set of options into a larger counterpart without additional learning. This is achieved through the combination of previously learned options. Our empirical evaluation focuses on options discovered for temporally extended exploration and the use of the successor representation to combine them. The results highlight important design decisions in option definition and showcase the synergy of different methods based on the successor representation, such as eigenoptions and the option keyboard.
[Abstract]