SR-AIF: sparse-reward robotics from pixels
Robots that learn from camera images rarely get told how well they are doing; usually the only signal is whether the task succeeded at the end. Active inference agents act to reach preferred states, but until this work they had mostly been studied where the full state is visible and the reward is dense.
SR-AIF has the agent learn its own preferences online. A contrastive prior preference model is trained on a signal that is positive along successful episodes and negative along failed ones, so it learns to imagine the next desirable state at every step. The policy is then trained with an actor-critic form of expected free energy that combines this goal signal with an exploration signal from the world model.
Main results
- On most tasks the agent converges earlier than DreamerV3 and two active inference baselines, and ends with a higher cumulative reward and success rate.
- On the two robosuite tasks, door opening and block lifting, it is the only agent of the four that succeeds.
- On Meta-World Push, where even the expert struggles, it reaches a 0.6 success rate against 0.1 for DreamerV3.