Abstract
Since the birth of arti?cial intelligence (AI), the ultimate goals of the ?eld have been to synthesize arti?cial agents that exhibit intellectual and problem-solving abilities com-parable or superior to those of human beings. One of the fundamental characteristics of such intelligence is the development of capabilities that enable an agent to make ef?-cient use of its interactions with the environment. This implies making use of not only the immediately available feedback from the environment, but also the outcomes of past experiences. For example, previously acquired knowledge of related tasks, or the memory of signi?cant events which have already occurred could be bene?cial while choosing an action in the present moment. From its past experiences, agents should also ideally be able to anticipate potential future tasks, and be roughly prepared for them by acquiring suitable priors, whenever possible. Equipping arti?cial agents with such capabilities would be a step towards replicating some of the observed character-istics of biological intelligence, such as the abilities to notice, anticipate and selectively remember experiences, and to subsequently plan and act in accordance.Reinforcement learning (RL) is a theoretically grounded approach for designing such agents, as it need not make any explicit assumptions regarding the dynamics of the agent or the environment. The learning is sequential, adaptive, and is based solely on scalar rewards resulting from the agent-environment interactions. However, RL algo-rithms are generally sample inef?cient, especially so in reward-sparse environments. As a result, RL training is usually carried out using simulations. Even RL applications deployed on physical platforms are typically (at least partially) pre-trained using sim-ulated agents and environments. In this dissertation, we adopt an RL framework, and introduce three methodologies directed at addressing issues related to the ef?cient use of an agent’s experiences: • The ?rst of these addresses the issue of task preparedness, or anticipation of possi-ble future tasks in RL. The idea is that an agent should be able make use of what-ever experiences occur, to identify ‘interesting’ regions of the state-space, and treat them as goal states of auxiliary tasks. Using off-policy algorithms, useful priors for these tasks can be learned in parallel, in addition to the value function associated with the primary task assigned to the agent. The identi?cation of ap-propriate auxiliary tasks allows agents to anticipate these potential future tasks by learning their corresponding value functions, at least partially, using whatever agent-environment interactions happen to take place.• The second methodology deals with the issue of intelligently reusing sequences of previous experiences (transitions) in order to accelerate learning, even when some of these tasks are reward-sparse in nature. We show that by storing and reusing selected sequences of experiences, it is possible to learn, not only from those ex-periences, but also from experiences which could have possibly occurred (which in reality, did not).• Lastly, we introduce an approach that is related to the issue of enabling an agent to leverage its previously acquired knowledge to make more informed exploratory actions while learning a new task. Simultaneously, this approach also enables the scalable storage of previously acquired task knowledge, avoiding redundancies that arise from learning multiple tasks that are very similar to each other. We posit that such an integrated knowledge storage and reuse mechanism would be very useful in the context of continual learning. The methodologies enlisted are validated empirically via simulations, and whenever possible, through experiments performed on the EvoBots, a micro-robotics platform that was developed to study the performance of a variety algorithms in the real world. The focus of this dissertation, however, is to bring about algorithmic improvements to the RL architecture, with an emphasis on improving the sample ef?ciency of learning. The hypothesis is that the intelligent reuse of past experiences, combined with the de-velopment of priors for potential future tasks could better inform the actions taken by an agent, thereby allowing new tasks to be learned with a fewer number of interac-tions with the environment. We posit that such improvements are especially useful in embodied applications, where time and energy costs of exploration outweigh the com-putational costs associated with making intelligent exploratory actions.