maze

RL in simulated physical environments

• the console · a generated apartment
• an episode, replayed step by step
• training runs · ppo and behaviour cloning
• the same scene document, path-traced
• a kitchen, path-traced in cycles
• what the agent sees, five tasks
• the rigged body and its walk cycle
• a large apartment at night

a simulated apartment where a general multimodal agent is given a body and an objective, to test whether the loop that works on a codebase (look, try something, read what came back, change your mind) also works on a physical environment it has never seen. a frozen agent/environment contract lets a learned policy, a symbolic planner and a frontier model drive the exact same body through the exact same interface, so the comparison means something. the environment, the task generator and the measurement came first; the agent comes second.

what the gap between a planner and a learned agent is made of

• reinforcement learning
• imitation learning
• dagger
• ppo
• behaviour cloning
• embodied agents
• world models
• test-time search
• evaluation
• procedural generation
• python

the questionWhich part of the gap is which

A planner that reads the scene graph solves 94.4% of the suite. A learned agent that sees 64 range rays, a memory map and a goal vector solves far less. The question was never whether these apartments are solvable, it was which part of that gap is observation, which is data, which is optimisation, and which is the benchmark itself.

Every comparison below is paired: the same scene and task handed to both arms, bootstrap intervals, a sign test. Where I got something wrong, the retraction sits next to the claim.

$$\mathrm{SPL}=\frac{1}{N}\sum _{i=1}^N{S}_i\frac{L_i^{\star }}{\max \left(L_i^{\star },L_i\right)}$$

failuresCounting instead of watching

It started with a replay: asked for the yellow bottle, the policy carried something else for 300 steps. One episode is an anecdote, so every eval episode went into exactly one bucket, crossing how it ended with what it had done.

Two readings reordered the work. Most of what RL added over its clone was fewer false DONEs, and the one bucket that grew from clone to RL was holding the wrong object. A tenth of the suite could not be solved by the oracle either, and that has to be netted out of any claim.

0:00 / 0:00

identificationIt could not tell objects apart

When a privileged policy and a learned one disagree wildly, suspect the observation before the learner. At the first grasp the policy took the target 33.6% of the time, against 33.0% for an agent picking blindly among what was in reach, and 74% of wrong grasps were a different type and colour. The goal named the object; nothing the agent saw carried either.

A sensor that labels what the rays hit got the agent to the right object more often and then it still picked blind. The fix was wiring: the goal and the per-ray labels only met after all 64 bearings were pooled, so I added a per-ray match against the target before pooling. Crowding the demonstrations did nothing on its own and only helped once the network could compute the comparison.

$$m_k^{\text{type}}=⟨{\iota }_k^{\text{type}},g^{\text{type}}⟩,m_k^{\text{colour}}=⟨{\iota }_k^{\text{colour}},g^{\text{colour}}⟩,k=1,\dots ,64$$

dead endsWhat did not work, and what I over-claimed

The goal vector never named a room, so 93% of the clone's go-to episodes ended in a false DONE. A room sensor fixed nothing: the longer-trained clone fixed it without the sensor by learning never to call DONE on that verb at all. Sensors a clone has to learn to ignore cost more than they give.

RL fine-tuning learned to quit. In failed episodes trying cost about 15 in reward against 2 for quitting, so I built a reward where quitting is never cheaper, and it scored 28.4% against the shipped reward's 28.6% with neither policy quitting. The critic and the starting clone had also changed in that pair, so the arithmetic is consistent with the behaviour but not shown to cause it, and I have left the claim visible.

$$\begin{array}{rl}{r}_t& =-0.005+[\Phi (s_{t+1})-\Phi (s_t)]+21_{\text{subgoal}}+101_{\text{success}}\\ & -0.051_{\text{collide}}-0.251_{\text{failed act}}-21_{\text{false DONE}}\\ \Phi (s)& =-\frac{1}{4}{d}_{\text{geo}}(s,g)\end{array}$$

$$\begin{array}{rl}\mathcal{L}(\theta )& =-{\mathbb{E}}_t\left[\min ({\rho }_t{\hat{A}}_t,\mathrm{clip}({\rho }_t,1-ϵ,1+ϵ){\hat{A}}_t)\right]\\ & +c_V{\mathcal{L}}_V-c_H\mathcal{H}[{\pi }_{\theta }]+\beta (t)\mathrm{KL}\left({\pi }_{\text{BC}}\Vert {\pi }_{\theta }\right)\\ \beta (t)& =0.8\cdot 2^{-t/600\text{k}}\end{array}$$

gazeLook at the path, then walk

The agent hit walls, and people do not steer that way: gaze lands a step or two ahead and the body follows as a reflex. So the action became one of the 64 bearings in view, turned into movement by a fixed reflex, which means walking into a wall is not an action the policy has.

It changed the failure mode reliably and the failure rate only marginally. The dominant bucket for every strong agent is now search, never reaching the first subgoal, and a per-step reflex cannot fix that.

free expertWhat you can do when the expert is free and the world resets

Robotics and LLM labs are bound by two facts: expert labels are expensive and the world does not reset. Here the expert is a scene-graph query and a reset is a copy of a document, so methods that are usually too costly become cheap.

DAgger, where the expert labels the states the learner actually reaches, took imitation alone past the best RL fine-tune. Rehearsal, trying the policy's top three actions in a snapshot before committing, doubled success on the same policy: not a deployable agent, but the ceiling on what planning inside a world model is worth, and a sign the proposals already contain the right action. Rollouts from resets also beat the learned critic at judging states.

0:00 / 0:00

$$\mathcal{D}\leftarrow \mathcal{D}\cup \left\{\right(s,{\pi }^{\star }(s)):s\sim d_{{\pi }_{\beta }}\},{\pi }_{\beta }=\beta {\pi }^{\star }+(1-\beta ){\pi }_{\theta }$$

$$a_t=\underset{a\in {\mathrm{top}}_3\pi (\cdot \mid s_t)}{\mathrm{argmax}}\frac{1}{2}\sum _{j=1}^2\sum _{k=0}^{24}r\left(s_{t+k}^{(j)}\right),s_t^{(j)}=\mathrm{restore}(s_t),a_t^{(j)}=a$$

$${\hat{V}}_K(s)=\frac{1}{K}\sum _{k=1}^K{G}_k(s),\mathrm{EV}=1-\frac{\mathrm{Var}[G-\hat{V}]}{\mathrm{Var}[G]}$$

where it standsThe benchmark, and what is still open

Before asking whether rankings survive a change to the simulator, I checked whether they survive resampling. Two halves of the same suite agree at Kendall tau 0.73 with 250 episodes each and 0.54 with 50, and 81% of neighbouring policies are within 2 points: the suite separates the top from the bottom and cannot order the middle.

So nothing that mattered here was a rank, and every claim above is a paired difference. What is left is mostly search, and a learned value or world model that buys a third of the rehearsal gain would be the biggest step available.

$$\tau =\frac{C-D}{\left(\genfrac{}{}{0ex}{}{n}{2}\right)}$$