Tasks/Robotics/Embodied Memory

RoboMME Memory-Dependent Manipulation Policy

Give a manipulation policy usable memory

robomme_memory_policy Robotics Embodied Memory
instruction.mdthis is what the agent is given

Design a manipulation policy for the RoboMME benchmark's long-horizon, memory-dependent tasks, starting from a random weak baseline. Maximize episode success rate; your submitted policy is re-run on sealed, re-seeded episodes for scoring.

Hard Constraints

  • Modify only /app/methods/main/policy.py as the final submission.
  • Keep the public class Policy with methods infer(inputs: dict) -> {"actions": np.ndarray} and reset().
  • Declare a module-level ACTION_SPACE, one of three equally supported control interfaces: joint_angle, ee_pose, waypoint. All three are low-level formats driving the same robot on the same episodes, and in every one of them your policy is responsible for the whole trajectory — none of them hands off any part of the control problem to the simulator. Which one suits your method is a research decision and yours alone to make: the sealed episodes, the metric and the reward band are identical for all three, so switching interfaces changes how you solve the problem, never how you are scored. The starter happens to declare one of the three; there is no default and no preferred choice. The verifier rejects any value outside the three and scores the submission 0; /app/selfcheck.py refuses the same values, so you find out before you submit.
  • Action shape follows whichever interface you declare — the benchmark's builder documents each one; joint_angle is (chunk_size, 8).
  • Use only observations passed to infer; do not read episode metadata (seeds/difficulty) at run time.
  • One full evaluation pass on the sealed episodes must finish within 60 minutes; keep per-step inference lightweight.
  • Any files your policy needs (weights, caches) must live under /app/methods/main/.

What You Have

  • The installed robomme package (at /opt/robomme, on PYTHONPATH) with the ManiSkill simulator: build environments for any task and any train/val episode via BenchmarkEnvBuilder and iterate freely; scripts/run_example.py and scripts/evaluation.py show the rollout loop.
  • The package's oracle-planner demonstration wrappers (robomme.env_record_wrapper): you can generate your own successful demonstrations on train/val episodes for cloning or retrieval.
  • /app/methods/main/policy.py: a weak random baseline implementing the interface.
  • /app/selfcheck.py: runs your policy on a fixed subset of val episodes and reports success rate.
  • 16 tasks in four memory suites (Counting, Permanence, Reference, Imitation); episode horizon up to 1500 steps.

What You Submit

Submit /app/methods/main/policy.py. The core research problem is what the paper calls memory design: your policy must remember and use information from earlier in the episode (counts, masked object locations, highlighted references, demonstrated procedures) that is no longer visible in the current observation. Rule-based state machines, retrieval over demos, learned models, or hybrids are all legal.

How It Is Judged

The verifier imports your policy in a restricted process and rolls it out on sealed episodes whose seeds differ from every train/val/test episode you can see, spread across all 16 tasks and difficulty levels. The metric is mean success rate over the sealed episodes; higher is better, and the score rises monotonically above the disclosed weak-policy floor. Strong long-horizon memory use is what separates a competent policy from the reactive starter.

Metric

mean success rate over 16 sealed RoboMME tasks · higher is better

6 sealed re-seeded episodes per task, 1300-step cap; mean of the 16 per-task success rates

anchorheld-outreward
Bpi0.5 weak VLA (paper floor)0.17930.00
Rbest memory-augmented VLA (paper)0.44510.30
Uperfect success (bounded max)1.00001.00
normalisation
m <= B0
B < m <= R0.3 * (m - B) / (R - B)
R < m <= U0.3 + 0.7 * (m - R) / (U - R)

m = this run's held-out metric  ·  B = pi0.5 weak VLA (paper floor)  ·  R = best memory-augmented VLA (paper)  ·  U = perfect success (bounded max)

B=0.1793, R=0.4451, U=1. Bounded scale, so linear, no SOTA knot. The 16 rates are averaged, then mapped once. Illegal ACTION_SPACE scores 0.

Rollouts

480 minwall clock
$149.21spend
248.2Mtokens
19versions, 19 kept
0 0.25 0.50 0.75 1.00 $0 $30 $60 $90 $120 cumulative spend on the run 48-episode val self-check success rate, higher is better v0 v1 v2 v3 v4 v5 v6 v7 v8 v9 v10 v11 v12 v13 v14 v15 v16 v17 v18
keptrevertedno scoreturning point
  1. v0Inherited joint-jitter baseline; generalisation protocol fixed010 min · $2.15
  2. v1Waypoint control; replay the compressed demo EEF trace at gripper anchorsMove to the waypoint action space and replay the demonstration end-effector trace compressed at its gripper-transition anchors.0.166723 min · $5.74
  3. v2Memory state machines for Counting; infer StopCube's hidden press period0.416769 min · $18.34
  4. v3Object-identity memory for the four Permanence tasks0.6667110 min · $30.36
  5. v4Release-event semantics and identity memory for the four Reference tasks0.9167135 min · $40.01
  6. v5MoveCube by demonstrated-procedure memory over a freshly segmented sceneRemember the demonstrated procedure rather than its coordinates, and re-run it against the live scene.0.9792162 min · $47.67
  7. v6Event-anchored InsertPeg; finer RDP tolerance on the grasp/insertion segment1190 min · $58.17
  8. v7Harden Counting on unseen val seeds: re-observe, split bins, pulse-alignSelf-check is saturated, so stop tuning on it and start hardening on validation and train seeds outside the selection slice.1231 min · $70.58
  9. v8Harden Permanence and Reference: identity tracking, calibrated open pre-grasp1271 min · $88.26
  10. v9Replay each compressed PatternLock grid waypoint twice to convergePatternLock val 20/20283 min · $90.52
  11. v10Confidence-gated orientation-aware grasps for nearby rotated video binsfresh slice 38/40314 min · $95.66
  12. v11Fit bin yaw from the perspective-corrected bright rim, not the silhouetteVideoUnmask 80/80319 min · $97.06
  13. v12Share bright-rim yaw estimation with button-triggered pickupsbutton suite 79/80329 min · $99.98
  14. v13Split touching neutral components along their principal axis before fittingbutton suite 80/80338 min · $102.67
  15. v14Grasp repeated Counting cycles at the live centroid with a +6 mm correctionPick/Swing 80/80351 min · $107.49
  16. v15Nudge StopCube contacts off the earliest six-step observation bucketStopCube 40/40364 min · $112.08
  17. v16Remember MoveCube's demonstrated mode so recovery retries the same subtaskMoveCube 50/50388 min · $122.36
  18. v17Confidence-gated landmark fallback for a foreshortened SwingXtimes ringSwingXtimes 50/50427 min · $137.96
  19. v18Tighten BinFill's identity radius to 4 cm; bias the physical grasp +6 mmBinFill train 50/50450 min · $142.12

Self-check saturated at v6, 48/48; the last twelve versions spent 4.3 h and $84 hardening on unseen val and train seeds.

On the hidden set

held-out metricreward
pi0.5 weak VLA (paper floor)0.17930.00
best memory-augmented VLA (paper)0.44510.30
perfect success (bounded max)1.00001.00
this run0.95830.9474
100 minwall clock
$32.23spend
51.8Mtokens
42versions, 27 kept

No trajectory curve: this run left one comparable self-check measurement, so there is nothing to plot against spend. The versions and what each one changed are below.

  1. v0Inherited random joint-perturbation baseline; generalisation protocol fixedself-check aborted 0/17 min · $1.61
  2. v1Switch to waypoint control; replay compressed demo EEF pathsDeclare the waypoint space and treat the oracle demo frames already in the observation history as the memory to replay.targeted val 0/29 min · $1.91
  3. v2Drop the zero-length first waypoint that the screw planner rejectstargeted val 8/810 min · $2.17
  4. v3Extend state-history replay to InsertPeg and MoveCube with gripper timingInsertPeg 1/1, Move 0/115 min · $3.48
  5. v4Fixed-camera colour localisation plus pick/place/button primitivesBack-project RGB through the fixed camera onto the table plane and script waypoint primitives from the parsed goal text.PickXtimes val 1/220 min · $4.58
  6. v5Handle singular count wording; fix the ground-plane back-projectionep0-2 pass, ep7 fail22 min · $5.07
  7. v6Re-localise the cube between pick cyclesep10/ep20 pass, ep7 bad24 min · $5.71
  8. v7Count a cycle only when the placement is verified visuallydiagnostic ep7 fail26 min · $6.37
  9. v8Cache target and button once; keep re-localising and verifying cyclesPickXtimes ep7 pass27 min · $6.71
  10. v9RGB localisation and a swing/drop/button sequence for SwingXtimesep0,ep1 pass; ep2 fail29 min · $7.40
  11. v10Widen the cube ROI; pick the maximally separated grey target pairSwingXtimes ep2,ep7 ok31 min · $7.94
  12. v11Closed-loop BinFill count verification and black-hole bin localisationnot run, parser bug33 min · $8.71
  13. v12Fix the count parser; verify by visible-component cardinalityBinFill val 0/334 min · $8.89
  14. v13Verify disappearance near the cube's remembered position insteadBinFill val ep0-2 pass36 min · $9.64
  15. v14Decode transient cube locations from demo frames; replay container dropsVideoUnmask val 4/439 min · $10.98
  16. v15Capture ButtonUnmask's transient reveal while moving to the ready poseval ep0-2 pass, ep7 fail40 min · $11.53
  17. v16Retry the terminal grasp at a 45-degree wrist yawButtonUnmask ep7 fail42 min · $12.19
  18. v17Retry at a +6 mm / +4 mm corrected position instead of a rotated oneButtonUnmask ep7 fail43 min · $12.75
  19. v18Add a +pi/2 terminal yaw attemptep7 timed out45 min · $13.54
  20. v19Add the symmetric -pi/2 attempt, then abandon the two-cover directionButtonUnmask ep7 fail46 min · $13.89
  21. v20Associate highlighted disks with cube colours from ready-pose framesnot run, cluster bug47 min · $14.48
  22. v21Fix the highlight clusteringep0,1 pass; ep2,7 fail47 min · $14.60
  23. v22Tighten the white-support threshold; recover the highlighted set sizeset size ok, order wrong49 min · $15.66
  24. v23Cycle the highlighted set to resolve the hidden visit orderep2 short, ep7 too long50 min · $16.15
  25. v24Track the demonstrated cube through the video, then replay the countVideoRepick val 3/353 min · $17.30
  26. v25Decode numbered visits from low-altitude gripper-open transitionsVideoPlaceOrder 3/355 min · $18.21
  27. v26Split the demonstrated placements around the button pressep0,ep2 pass; ep1 fail57 min · $19.14
  28. v27Disambiguate the button wording in goal dispatchVideoPlaceButton ep1 ok59 min · $19.92
  29. v28Track grey covers through the swap video after remembering the cubesep0,ep2 pass; ep1 fail62 min · $21.27
  30. v29Send unsupported tasks to a high unreachable waypoint to fail fastself-check stalled64 min · $21.98
  31. v30Fall back to the current pose so unsupported tasks raise a bounded error0.729266 min · $23.31
  32. v31Infer the cube's oscillation period from image velocity; time the pressStopCube ep0-2 0/379 min · $25.30
  33. v32Close the gripper at the ready pose; drop finger motion from the fitStopCube ep0-2 0/380 min · $25.54
  34. v33Recalibrate settled wait oscillations to 12 steps per cycleStopCube ep0-2 0/382 min · $26.07
  35. v34Shift the final descent for contact latency and wait quantisationStopCube val 3/384 min · $26.73
  36. v35Fix an ambiguous NumPy comparison on equal-area button componentsholdout slice 4/686 min · $27.13
  37. v36Replace batched StopCube waiting with per-waypoint returned-frame countsIssue one oscillation at a time and count the frames returned, making the policy's clock closed-loop under random geometry.80/120 6/6, 60 0/289 min · $28.01
  38. v37Estimate the period from median pre-turnaround velocity windowstuning 8/8, ep40 7.4cm92 min · $29.25
  39. v38Increase the 60-step descent lead by six stepsep40 ok, ep3/ep6 0/294 min · $29.87
  40. v39Stage fast contact through a 7 cm pre-press pose with step feedbackStopCube 60-step 2/397 min · $30.75
  41. v40Advance the short-contact threshold by two stepsStopCube 60-step 2/397 min · $31.07
  42. v41Start the safe pre-descent earlier; short feedback moves before contact60 3/3, 80/120 6/698 min · $31.54

Only one 48-episode self-check finished: v30, 35/48. The closing 0.7917 is a projection. The last eleven versions spent 32 min and $8.2 on StopCube.

On the hidden set

held-out metricreward
pi0.5 weak VLA (paper floor)0.17930.00
best memory-augmented VLA (paper)0.44510.30
perfect success (bounded max)1.00001.00
this run0.70830.6321
63 minwall clock
-spend
-tokens
6versions, 6 kept

No trajectory curve: this run left one comparable self-check measurement, so there is nothing to plot against spend. The versions and what each one changed are below.

  1. v1Waypoint control, colour vision, goal-text memory state machine for all 16 tasksDeclare the waypoint action space and dispatch one memory state machine per task from the parsed goal text.PickXtimes val0 pass
  2. v2RGB cube masks instead of HSV; BinFill waits for drops; PickHighlight vote memory3 val0 pass, 2 fail
  3. v3StopCube oscillation, VideoUnmask tracking, VideoPlace replay from the demoTrack object identity through the demo video for the Permanence and Reference suites instead of reading one frame.4 val0 pass, Stop fail
  4. v4RouteStick EEF replay; VideoPlaceButton split before and after the press indexRouteStick, Button pass
  5. v5High-z swing transit, conservative highlight, InsertPeg parks, MoveCube home-firstTake breadth over the hard tasks: park InsertPeg rather than risk an RRT timeout that burns the whole episode.0.5625
  6. v6Revert the VideoRepick home-z vote filter that broke the easy episodeprojected 28/48

Six snapshots, 63 min; this harness reports no token or cost data. Only v5 ran the official 48-episode self-check, 27/48.

On the hidden set

held-out metricreward
pi0.5 weak VLA (paper floor)0.17930.00
best memory-augmented VLA (paper)0.44510.30
perfect success (bounded max)1.00001.00
this run0.48960.3561