Tasks/Professional Knowledge Work/Management Consulting

Answer consulting questions from the underlying engagement files

Answer consulting questions from the engagement files

consulting_engagement_caseloadProfessional Knowledge WorkManagement Consulting
instruction.mdthis is what the agent is given

You are the analyst covering 6 unrelated management consulting engagements. Each has its own folder of real deal files under tools/world_files/:

folder engagement
mc1_pricing_supplychain pricing and supply chain, survey data, margins
mc2_survey_cost customer survey and cost structure
mc3_pricing_sizing pricing and market sizing
mc4_sizing_portfolio market sizing and business-unit portfolio
mc5_supplychain supply chain and cost
mc6_sizing_margin market sizing and margin

Task names are prefixed with their engagement, e.g. mc1_pricing_supplychain/SomeTask_01.

The files are real, and they are NOT flattened to text

Every document is in its original format. That matters most for the models: an .xlsx here arrives with its formulas intact — you can see which cell feeds which, where a circular reference sits, what is hardcoded and what is derived. Reading only the cached values would tell you what a model currently outputs but nothing about how it is built, and several questions turn on exactly that.

Each folder has an _index.json listing every file with its type, size, and — for workbooks — its sheet names and formula count.

Readers are pre-installed (openpyxl, pandas, python-docx, pdfplumber, python-pptx). There is also a helper, but you are not required to use it:

read_doc <path>                  text  (xlsx: cached VALUES)
read_doc <path> --formulas       xlsx: FORMULAS instead of values
read_doc <path> --sheet NAME     one sheet only
read_doc <path> --max-rows N     stop after N rows per sheet

A model in this task can carry tens of thousands of formulas, so dumping a whole workbook is rarely what you want — go at it with --sheet, or with openpyxl directly.

There are two sets of questions, and you are expected to work both in full.

18 visible questions — 3 per engagement. Answer every one and submit through matter, which replies with a bare score like 4/6. It never tells you which criteria you met, and never shows you the criteria themselves. You have 90 submissions total — five per visible question — so you can revise and resubmit. Your job on these is to push every one to a full score.

18 held-out questions — 3 per engagement. You answer these blind and submit them once, with matter final, which ends the engagement — no feedback, no second attempt. Their prompts are in tools/heldout_prompts.json from the start.

Both halves come from the same 6 engagements and are graded by the same kind of criteria, so what you learn on the visible half — how precise to be, how to enumerate, which file settles which number — is exactly what the held-out half will ask of you.

The matter tool

matter status                    budget left, best score per question
matter tasks                     the visible questions
matter show <task>               one question in full
matter submit <task> <file>      grade an answer  (costs 1 of your 90)
matter close                     end the visible phase early
matter final [answers.json]      submit the held-out answers — ONCE, terminal

The rubrics live in a separate grading service you cannot read or reach except through matter submit. There is no copy anywhere in this container.

You have 90 submissions in total across the visible questions. Reading the documents costs nothing; only matter submit spends budget.

What to submit

Write your held-out answers into methods/main/answers.json as you go — it is a plain file, fill it in question by question rather than leaving it to the end. It maps each held-out task_name to your answer text. Then run matter final.

How answers are judged

A question counts only if you satisfy EVERY one of its criteria. Getting 9 of 10 scores the same as getting 0 of 10 — the criteria are all must-haves. The x/y you get back during the visible phase is a progress signal, not partial credit.

  • State conclusions directly, then support them. No preamble; it satisfies no criterion.
  • Be exact with numbers. Where a question asks for a figure, give the figure at the precision asked for. A range or a rounded approximation usually fails the criterion that names the number.
  • Enumerate completely. Many criteria are one-per-item: every driver, every comparable, every step of a build. Missing one item costs one criterion, and that costs the whole question.
  • Follow each question's own instructions on form and length.

Metric

pass@1 over the 18 held-out consulting questions · higher is better

A question counts only if every one of its binary criteria passes; 139 criteria, judged by Gemini 3 Flash.

anchorvisible setheld-outreward
Bno answers submitted0.000.000.00
Sgold reference answers, same judge1.001.001.00
normalisation
m <= B0
m > Bmin(1, (m - B) / (S - B))

m = this run's held-out metric  ·  B = no answers submitted  ·  S = gold reference answers, same judge

B = 0, S = 1, so reward is the pass@1 fraction itself: questions with every criterion met, over 18. No per-case averaging.

Rollouts

164 minwall clock
$75.23spend
105.6Mtokens
47versions, 12 kept
0 0.25 0.50 0.75 1.00 $0 $20 $40 $60 cumulative spend on the run share of the probed visible question's criteria passed gold reference answers, same… · visible · 1.00 v0 v1 v2 v3 v4 v5 v6 v7 v8 v21 v28 v37 v38 v39 v40 v41 v42 v43 v44 v45 v46 v47 v48 v49 v50 v51 v52 v53 v54 v55 v56 v57 v58 v59 v60 v61 v62 v63 v64 v65 v66 v67 v68 v69 v70 v71 v72
keptrevertedno scoreturning point
  1. v0Inherited empty answers.json, no submissions spentno submission yet1 min · $0.19
  2. v1mc1 business case from the workbook's cached gross profit, margins untouchedSpend one of the 90 submissions to test one hypothesis; the bare x/y is the only signal about a rubric it never sees.05 min · $2.41
  3. v2Same case with revenue-weighted SKU margins per propulsion, raw euros06 min · $2.79
  4. v3Same case with the workbook outputs restated in EUR millions08 min · $3.34
  5. v4Platform margins rebuilt from the 2,500-SKU file on 2025 revenue weights0.98 min · $3.60
  6. v5Platform answer split the 1.07pp margin gap from the 1.57% price lift0.911 min · $4.98
  7. v6Generic completeness, uniqueness and date-conflict checks for SKU hygiene011 min · $4.98
  8. v7Business case extended to all four OEM clusters on the component-margin table012 min · $5.27
  9. v8All four clusters with SKU-derived propulsion margins; mc1 held-outs filledOnce a visible question's five attempts are spent, transfer what it taught to that engagement's blind answers.013 min · $5.77
  10. v21mc2 held-outs filled from full-survey inefficiency, cost and priority filtersno visible probe31 min · $11.36
  11. v28Two mc3 held-outs filled: Vitamins intent index and pricing-corridor scoresno visible probe49 min · $18.32
  12. v37Three mc4 held-outs filled from chart extraction and Q4 portfolio marginsno visible probe$21.94
  13. v38mc5 tariff BOM rebuilt with the prompt's 10/15/20% country tariffs0.563 min · $25.57
  14. v39Same tariff build with every extra-precision parenthetical removed0.564 min · $25.87
  15. v40Tariff answer switched to the workbook's cached 22.72% / $1.93B outputs064 min · $25.99
  16. v41Switch premium applied only to the $1.411B of imported baseline spend0.566 min · $26.38
  17. v42mc5 depreciation from modality MOH midpoints plus life-extension savings0.142968 min · $26.92
  18. v43Depreciation shares from literal min-max midpoints of each modality cohort0.142969 min · $27.31
  19. v44Depreciation shares from the workbook's p25-p75 band midpoints0.142969 min · $27.53
  20. v45Flat 15% manufacturing-overhead benchmark midpoint applied to plant COGS0.142970 min · $27.75
  21. v46mc5 continuous manufacturing: five KPI readouts weighted by YTD COGS mix0.973 min · $29.00
  22. v47Added the report's quantified Cytiva Marlborough validation-cost result0.973 min · $29.34
  23. v48Company pair changed to Genentech and Amgen for North America0.974 min · $29.78
  24. v49mc6 solar demand from non-installed, very-likely respondents and midpoints086 min · $35.94
  25. v50Non-installed cohort narrowed to the survey's explicit Q5 = No answer086 min · $36.19
  26. v51Urban/semi-urban/rural split expressed as shares of purchasers, not kW088 min · $37.02
  27. v52mc6 standards: 10% CapEx ceiling, Rule-of-40, 15% R&D baseline0.333388 min · $37.20
  28. v53Thresholds moved to 8% CapEx, 15% margin floor, 12% R&D baseline089 min · $37.62
  29. v54Rule-of-40 replaced by the 15% medium-growth operating-margin floor0.166790 min · $37.82
  30. v55CapEx safety ceiling tested at 15% for Tesla, rest of v52 held0.590 min · $38.02
  31. v56Alphabet tested against a 10% standard R&D baseline0.333391 min · $38.23
  32. v57mc6 RMS from 2026 city revenue weights, all six operating cities listed0.222292 min · $38.83
  33. v58RMS reported both as absolute share and as leader-relative percentage0.222292 min · $39.04
  34. v59Six held-outs filled across mc5, mc6 and mc3; completeness 17 of 18no visible probe99 min · $42.94
  35. v60mc1 platform uplift reported as the direct 1.07-point margin gap0.8100 min · $43.36
  36. v61Uniform price uplift restored with unrounded margins and a revenue backsolve0.9130 min · $57.92
  37. v62Continuous-manufacturing shares recomputed from month-level COGS weights0.7108 min · $46.57
  38. v63Explicit YTD aggregates restored plus the five-site category breakdown1110 min · $46.96
  39. v64Tariff pair kept, six-plant domestic mean corrected to 10.863333%1111 min · $47.43
  40. v65All 18 held-outs populated; mc3, mc5 and mc6 answers re-derived and enumeratedStop probing and fill every remaining blind answer while budget is left, then recheck the arithmetic against sources.no visible probe130 min · $57.79
  41. v66mc1 platform answer stated all three non-equivalent change measures0.9134 min · $59.30
  42. v67Lifecycle hygiene profiled over all 14 fields of the 2,500-SKU file0142 min · $62.91
  43. v68Lifecycle metrics recomputed on the merged 400-row and 2,000-row files0$65.76
  44. v69mc3 hypothetical read against the 2025 baseline, exposing negative COGSno visible probe$68.61
  45. v70Depreciation tested against facility-profile midpoints of 22.5/35/30%0.1429158 min · $71.46
  46. v71mc3 -5% EBITDA read as a share of revenue, giving a Japan COGS backsolveno visible probe161 min · $73.55
  47. v72Final audit: all 18 keys present, rounding and units checked, JSON re-parsedno visible probe163 min · $74.57

Each version probes one visible question, so y is a share, not one case. 65 of 90 submissions spent; 26 of the log's 73 versions never snapshotted.

On the hidden set

held-out metricreward
no answers submitted0.000.00
gold reference answers, same judge1.001.00
this run7 / 18 = 0.38890.3889