maze

RL in simulated physical environments

simulated apartments where an agent gets a body and an objective in English. the question is whether the loop that works on a codebase (look, try something, read the result, adjust) also works in a physical space it has never seen. a learned policy, a symbolic planner and a frontier model all drive the same body through the same interface, so their results compare directly. the environment, the task generator and the measurements were built first, and the agent second.

where the gap between a planner and a learned agent comes from

reinforcement learning · imitation learning · dagger · ppo · behaviour cloning · embodied agents · world models · test-time search · evaluation · procedural generation · python

the questionWhere the gap comes from

An oracle planner that reads the scene graph solves 94.4% of the tasks. A learned agent that only sees 64 range rays, a memory map and a goal vector solves far fewer. The apartments are clearly solvable, so the question is what the gap is made of: how much comes from what the agent can observe, how much from the data, how much from training, and how much from the benchmark itself.

Every comparison here is paired. Both agents get the same scene and the same task, with bootstrap intervals and a sign test. Where I got something wrong, the correction sits next to the original claim.

$$\mathrm{SPL}=\frac{1}{N}\sum _{i=1}^N{S}_i\frac{L_i^{\star }}{\max \left(L_i^{\star },L_i\right)}$$

• oracle94.4%
• RL over gaze30.2%
• DAgger, round 229.4%
• RL fine-tune28.6%
• longer clone24.0%
• first PPO14.6%
• first clone12.0%
• random1.0%

failuresSorting every failure into a bucket

It started with a replay. Asked for the yellow bottle, the policy carried something else around for 300 steps. One episode is an anecdote, so I sorted every evaluation episode into exactly one bucket, by how it ended and what the agent had done.

Two results changed the plan. Most of what RL added over the cloned policy was fewer false DONEs, and the only bucket that grew from clone to RL was holding the wrong object. A tenth of the suite can't be solved by the oracle either, so that has to come out of any claim.

0:00 / 0:00

• timed out, wrong object26.0%
• timed out, did nothing26.0%
• success17.6%
• false DONE, did nothing11.8%
• oracle can't solve it10.8%

identificationIt couldn't tell objects apart

When a privileged policy and a learned one disagree this much, check the observation before the learner. On its first grasp the policy took the target 33.6% of the time, against 33.0% for an agent grabbing blindly among whatever was in reach, and 74% of wrong grasps were a different type and colour. The goal named the object, but nothing the agent saw carried that information.

A sensor that labels what each ray hits got the agent to the right object more often, but it still picked blindly. The problem was in the network: the goal and the per-ray labels only met after all 64 bearings had been pooled. Adding a per-ray match against the target, before pooling, fixed it. Crowding the demonstrations did nothing on its own, and only helped once the network could make the comparison.

$$m_k^{\text{type}}=⟨{\iota }_k^{\text{type}},g^{\text{type}}⟩,m_k^{\text{colour}}=⟨{\iota }_k^{\text{colour}},g^{\text{colour}}⟩,k=1,\dots ,64$$

• sensor + match+19.0 p<.001
• sensor + match + crowd+16.9 p<.001
• sensor only+9.5 p=.02
• crowd + sensor+3.2
• crowded demos only-1.1

dead endsWhat didn't work, and what I overclaimed

The goal vector never named a room, so 93% of the clone's go-to episodes ended in a false DONE. A room sensor didn't fix it. The longer-trained clone fixed it without the sensor, by learning never to call DONE on that verb. A sensor the policy has to learn to ignore costs more than it gives.

RL fine-tuning learned to give up. In failed episodes, trying cost about 15 in reward and quitting cost about 2. I built a reward where quitting is never cheaper, and it scored 28.4% against 28.6% for the shipped reward, with neither policy quitting. But the critic and the starting clone also changed between those runs, so the numbers fit the explanation without proving it. The claim stays up, with that caveat.

• room sensor+7 goto, -10 find
• target memorybelow control
• route plannerbelow control
• crowded demos-0.3 alone
• dense gaze headbest loss, 22.2%
• greedy decoding12.8 vs 12.6

gazeLook where to go, then walk

The agent kept walking into walls. People don't steer like that: we look a step or two ahead and the body follows by reflex. So the action became one of the 64 bearings in view, turned into movement by a fixed reflex, and walking into a wall stopped being something the policy could choose.

It changed how the agent fails much more than how often. For every strong agent, the biggest failure bucket is now search, never reaching the first subgoal, and a per-step reflex can't fix that.

left / right / forward	gaze target
22.6% success	24.6% success
7.6% collision frames	4.7% collision frames
9 episodes wedged	0 episodes wedged
31.8% never started	27.6% never started

free expertWhen the expert is free and the world resets

Robotics and LLM labs work under two constraints: expert labels are expensive, and the world doesn't reset. Here the expert is a scene-graph query and a reset is a copy of a JSON document, so methods that are usually too expensive become cheap.

DAgger, where the expert labels the states the learner actually visits, took imitation alone to 29.4% in two rounds, past the best RL fine-tune. Rehearsal tries the policy's top three actions in a copy of the world before committing to one, and it doubled success on the same policy, from 38.7% to 80.7% on the first 150 episodes. It isn't deployable, but it shows what planning inside a world model could be worth, and that the right action is usually already in the policy's top three. Rollouts from a reset also judged states better than the learned critic, explaining 0.42 of the variance against 0.17.

0:00 / 0:00

where it standsHow far to trust the benchmark, and what's open

Before asking whether rankings survive a change to the simulator, I checked whether they survive resampling. Two halves of the same suite agree at Kendall tau 0.73 with 250 episodes each, and 0.54 with 50. 81% of neighbouring policies are within 2 points of each other. The suite can separate the top from the bottom, but it can't order the middle.

So no claim here rests on a ranking; every one is a paired difference. Most of what's left is search, and a learned value function or world model that recovers even a third of the rehearsal gain would be the biggest step available.

close to settled	open
identification, with matching	the gap is now search
wrong object down from 39% to 4%	does it survive RL
DAgger beats RL fine-tuning	is the weak critic the shared trunk
mid-table rankings are noise	how much a learned world model buys