End-to-End Neuro-Symbolic Reinforcement Learning with Textual Explanations

End-to-End Neuro-Symbolic Reinforcement Learning with Textual Explanations
ICML 2024 (Spotlight, top 3.5%)

¹Peking University
²Beijing Institute for General Artificial Intelligence (BIGAI)

Abstract

Neuro-symbolic reinforcement learning (NS-RL) has emerged as a promising paradigm for explainable decision-making, characterized by the interpretability of symbolic policies. NS-RL entails structured state representations for tasks with visual observations, but previous methods cannot refine the structured states with rewards due to a lack of efficiency. Accessibility also remains an issue, as extensive domain knowledge is required to interpret symbolic policies. In this paper, we present a neuro-symbolic framework for jointly learning structured states and symbolic policies, whose key idea is to distill the vision foundation model into an efficient perception module and refine it during policy learning. Moreover, we design a pipeline to prompt GPT-4 to generate textual explanations for the learned policies and decisions, significantly reducing users' cognitive load to understand the symbolic policies. We verify the efficacy of our approach on nine Atari tasks and present GPT-generated explanations for policies and decisions.

Approach Overview

INSIGHT consists of three components: a perception module, a policy learning module, and a policy explanation module. The perception module learns to predict object coordinates using a frame-symbol dataset distilled from vision foundation models. The policy learning module is responsible for learning coordinate-based symbolic policies. In particular, to address with the limited expressiveness of object coordinates, it uses a neural actor to interact with the environment. The policy explanation module can generate policy interpretations and decision explanations using task description, policy description, and values of object coordinates.

Refer to our paper for Table 1, Table 2, and Table 4, we quantitatively demonstrate INSIGHT's improvements in return and predicting object-related coordinates. We include demos for segmentation and interpretation here.

The Impact of Policy Learning on Segmentation

Here are the segmentation results for nine Atari games, before and after policy learning. It has been observed that the accuracy of policy-irrelevant objects decreases, whereas the accuracy of policy-related objects increases.

Videos before and after policy learing.

Freeway

Seaquest

BeamRider

Breakout

Enduro

MsPacman

Pong

Qbert

SpaceInvaders

Policy illustration

Here is an example for language explanation for Pong. Left: interpretations for a learned policy. The interpretations identify influential input variables and summarize triggering patterns of actions. Right: explanations for an action taken at a state. The four images located at the bottom illustrate the state. The motion of the ball and the opponent's paddle are deduced from input variables, which are used for supporting explanations of actions.

You can view all prompts and LLM responses in the two buttons below.

Policy Interpretation

Decision Explanation

BibTex

@article{luo2024insight,
    title={End-to-End Neuro-Symbolic Reinforcement Learning with Textual Explanations},
    author={Luo, Lirui and Zhang, Guoxi and Xu, Hongming and Yang, Yaodong and Fang, Cong and Li, Qing},
    journal={International Conference on Machine Learning (ICML)},
    year={2024}
}