Causal Inference as an Inductive Bias in Reinforcement Learning

This project explored how causal inductive biases—specifically interventions and counterfactual reasoning—can enhance generalization in reinforcement learning (RL). We conducted a systematic comparison between model-free RL agents and causal RL agents using the CausalCF framework within the CausalWorld robotic manipulation simulator.

Agents were trained on a goal-conditioned robotic picking task and evaluated across 12 domain-shift protocols (P0–P11), which systematically altered environment variables such as block mass, pose, goal position, and friction. We used fractional success as the primary metric—quantifying overlap between the final and goal block configurations.

Our study tested four configurations:

1. Baseline SAC (no causal reasoning)
2. CausalCF (causal representation without interventions)
3. CausalCF + Intervene (with both intervention and counterfactual training)
4. Transfer-CausalRep (transfer of causal modules from picking to pushing task)

Results showed that causal agents significantly outperformed SAC in moderate distribution shifts (e.g., changes in mass or goal pose). CausalCF + Intervene achieved higher robustness, particularly under moderate structural variation. Additionally, the Transfer-CausalRep agent—trained solely on picking but evaluated on pushing—retained competitive performance, supporting the hypothesis that causal representations are partially transferable across tasks with shared structure.

However, all methods—causal and non-causal—saw degraded performance under severe domain perturbations (e.g., combined variation in mass, friction, and pose), pointing to limits of current causal RL approaches in highly randomized settings.

This project contributed a controlled empirical evaluation of causal RL under domain shift and task transfer, reinforcing the potential of structural causal models and counterfactual reasoning to improve robustness in RL-based robotic systems.

The full report is available here.