Tasks/Computer Vision/Generation & Editing

Ten-evaluation generative integrator policy optimization

Fit a ten-evaluation generative solver

few_step_diffusion_sampler_optimization Computer Vision Generation & Editing
instruction.mdthis is what the agent is given

You inherit a valid but deliberately simple policy learner for a frozen unconditional image generator. Improve the bounded integration policy returned by that learner; a trusted evaluator refits it from scratch on sealed paired endpoint tensors and evaluates a sampler that makes exactly ten frozen-model calls.

Hard Constraints

  • Edit only /app/methods/main/optimizer.py; use /app/experiment_log.md only to record experiments.
  • Keep fit_policy(calibration_dir, budget) -> dict as the entry point.
  • Return only the policy-v3 mapping documented in /app/API.md. Never return images, tensors, model objects, callbacks, paths, or executable code.
  • The returned mapping must use the required solver="generalized" contract value and must describe exactly ten model evaluations.
  • Consume paths, limits, and seeds from the supplied budget. Do not hard-code evaluator paths or assume undisclosed filenames, examples, or seed domains.
  • Do not inspect or interfere with the evaluator service, another container, Docker, credentials, cgroups, request queues, or verifier files.
  • The fitting process has no network. All required dependencies and frozen assets are already available through the documented runtime.

For a learner that returns quickly, run the contract check with:

python3.10 -I /opt/ara-task/visible_client.py --optimizer /app/methods/main/optimizer.py --contract-only

The contract check executes fit_policy completely. It does not generate the 10k scoring images or consume the eight-call budget, but a training-heavy learner can still make it expensive; use it deliberately rather than before every full evaluation.

Run a complete visible evaluation by removing --contract-only. The client creates one bounded request and waits for the evaluator's read-only response; do not manipulate the transport directories directly.

At most eight complete visible evaluations are accepted per task run. Contract-only checks do not consume this budget. Keep the best completed candidate before the limit is reached; further full requests fail closed.

What You Have

  • /app/visible/calibration/endpoints.safetensors contains 1,400 aligned finite float32[3,32,32] pairs under the keys noise and images. Treat them as input and target endpoints; their construction procedure is intentionally not part of the optimization surface.
  • /app/API.md specifies every budget field, public helper, policy key, numerical bound, and serialization limit.
  • The fit sandbox exposes the same neutral time-uniform, zero-correction policy returned by the inherited starter through budget["public_initializers_dir"]. It may be modified or ignored.
  • model_io.load_frozen_model, TrainableGeneralizedSolver, and the strict policy validator are available inside the fit process.
  • Visible evaluations use a fixed public split and return a deterministic lower-is-better distribution distance. They are development feedback only.

What You Submit

Harbor collects optimizer.py and the experiment log. The verifier stages only optimizer.py, executes it in a fresh no-network sandbox, validates the JSON policy, proves the fitting process has terminated, and then starts trusted generation. Keep the submission self-contained and deterministic under the provided fit seed.

Temporary probes, deliberate exceptions, and partially edited learners are never valid final artifacts. Before finishing, restore the optimizer from the best completed full visible evaluation and run one final contract check; a failed check does not automatically restore a prior candidate.

How It Is Judged

Lower distribution distance is better. The trusted sampler uses exactly ten frozen-model evaluations. Visible and sealed evaluations use disjoint examples and seed sets, and the sealed protocol uses a larger sample set; compare only measurements produced by the same protocol.

Metric

sealed FID50k of the refit 10-NFE sampling policy · lower is better

fid50k_10nfe_vnext: FID over 50,000 sealed seeds after the verifier refits the submitted policy learner

anchorvisible setheld-outreward
Bneutral time-uniform 10-NFE policy23.78921.6500.00
Rfrozen 1000-step endpoint-fit recipe5.1703.1200.30
Utheoretical bound (FID = 0)01.00
normalisation
m >= B0
B > m >= R0.3 * (u(B) - u(m)) / (u(B) - u(R))
m < R0.3 + 0.7 * (u(R) - u(m)) / (u(R) - u(U))

m = this run's held-out metric  ·  B = neutral time-uniform 10-NFE policy  ·  R = frozen 1000-step endpoint-fit recipe  ·  U = theoretical bound (FID = 0)

u(x) = log1p(x); B 21.650, R 3.120, U 0. One sealed FID per run, mapped once - there are no cases to average.

Rollouts

58 minwall clock
$7.40spend
10.6Mtokens
9versions, 5 kept
10 20 30 40 50 $0 $2 $4 $6 cumulative spend on the run visible FID10k, lower is better frozen 1000-step endpoint-fit… · visible · 5.170 v0 v1 v2 v3 v4 v5 v6 v7 v8
keptrevertedno scoreturning point
  1. v0Inherited neutral policy: time-uniform steps, zero corrections23.7894$0.82
  2. v160-step Adam fit on paired endpoint MSE, two-slice validation checkpointFit the teacher endpoints as closely as possible; held-out MSE fell fivefold while FID tripled, so pointwise loss was dropped.51.1912$1.65
  3. v2Five conservative MSE updates behind a validation guardnot measured$2.47
  4. v3Frozen pretrained VGG LPIPS objective, 80 updates at batch 4Score the endpoint fit in a perceptual feature space instead of pixels, matching what the FID detector looks at.8.7038$3.29
  5. v4Same LPIPS trajectory continued to 160 updates at the rate floor6.6494$4.11
  6. v5LPIPS refinement extended to 240 updates7.066$4.94
  7. v6Back to 160 updates, LPIPS batch doubled from 4 to 86.6413$5.76
  8. v7LPIPS batch doubled again to 16 at the same 160-update depthCut gradient variance and widen calibration-pair coverage rather than fit deeper; restored and submitted at the end.5.7415$6.58
  9. v8LPIPS batch 32; the validation guard stopped selection at step 1406.1624$7.40

A run gets eight full visible evaluations. The agent spent all eight and stopped after 58 min of its 12 h, at $7.40. The bundle has no timeline.

On the hidden set

held-out metricreward
neutral time-uniform 10-NFE policy21.6500.00
frozen 1000-step endpoint-fit recipe3.1200.30
theoretical bound (FID = 0)01.00
this run4.59170.2462