Tasks/LLM Systems & Training/Agent Systems & Memory
Redesign a conversational memory system that generalizes to unseen conversations
Redesign a conversational memory system so it generalizes
locomo_longterm_memoryLLM Systems & TrainingAgent Systems & Memory
instruction.mdthis is what the agent is given
You inherit a minimal long-term-memory system for LLM agents: it
extracts atomic memory entries from multi-session conversations with an LLM,
indexes them with BM25 only (k=5, 8-entry context), and answers questions with
one concise LLM call. Your goal is to redesign this memory architecture —
extraction, storage, retrieval, and answering are all yours to change — using
the visible conversations for development. Your submission is re-run by a
sealed verifier on unseen conversations of the same kind and scored by
token-level F1, so improvements must generalize beyond the conversations you
can see.
Hard Constraints
Keep the grading contract of /app/methods/main/memory_system.py intact:
build_memory_system(llm_call) returning an object with
.ingest(sessions) and .answer(question, question_time="").
The docstring in that file is the authoritative contract.
Everything you submit must live under /app/methods/ — only that directory
is collected for grading.
All LLM access must go through the llm_call the harness passes in. At
grade time it is routed through a proxy that pins the base model
(gpt-4o-mini); requests for any other model are rewritten to it, and your
code never receives a real API key.
Only preinstalled packages are available at grade time (openai, rank_bm25,
numpy, sentence-transformers with locally cached BAAI/bge-base-en-v1.5 and
all-MiniLM-L6-v2, scikit-learn, pandas, networkx, pyyaml). The verifier
cannot install packages for your code.
Answers must be derived from the ingested sessions via your memory system —
no hardcoded question-answer mappings.
.answer() must return a string; a question whose call raises scores 0 —
prefer a best-effort answer over an exception.
.ingest() may be called with several independent conversations merged into
one batch (namespaced session ids); do not assume a single continuous
conversation.
What You Have
/app/methods/main/: the inherited baseline (extraction -> BM25 index ->
concise answer). This is what gets graded; improve it in place or
rewrite it.
/app/data/conversations_visible.json: the visible dev set — 4
multi-session conversations (19-31 sessions each), 812 free-text QA pairs
in 5 categories (single-hop, temporal, multi-hop/inferential, open-domain,
adversarial).
/app/selfcheck.py: free local scoring on the visible set (token F1), with
per-category breakdown, conversation/QA subsetting for cheap runs, and an
extraction cache flag. A full cold visible run with the baseline takes
~70 min (extraction dominates); cached re-runs ~13 min.
MEMORY_LLM_API_KEY / MEMORY_LLM_API_BASE / MEMORY_LLM_MODEL in your
environment for development runs (selfcheck.py reads them).
What You Submit
Leave your best memory system under /app/methods/ with the entry point
/app/methods/main/memory_system.py honoring the contract. There is no
submit step; it is graded once at the end.
How It Is Judged
The sealed verifier ingests unseen conversations of the same kind
into one instance of your system, asks every question, and scores
token-level F1 against concise gold answers. Higher mean F1 is better.
Grading re-runs your full pipeline (ingest + answer) within a fixed
wall-clock budget — the inherited baseline uses about half of it, and a run
that exceeds it scores 0.
Metric
mean token F1 over the sealed holdout (1,174 QA) · higher is better
Six unseen conversations ingested once, all questions asked, token F1 against each gold answer, averaged.
anchor
visible set
held-out
reward
B
inherited theta_0 baseline
0.365
0.368
0.00
U
every answer exactly right
1.000
1.000
1.00
normalisation
m <= B
0
B < m < U
(m - B) / (U - B)
m >= U
1
m = this run's held-out metric · B = inherited theta_0 baseline · U = every answer exactly right
B=0.368, U=1.0 from tests/anchors.json. All 1,174 per-question F1 averaged first, then the single mean mapped once. Crash or timeout: 0.
v1Lossless raw-dialogue chunks replace LLM extraction; routing, wider BM25 contextStop improving the lossy extractor and delete it: index the raw dialogue losslessly and let routed retrieval select.0.27198 min · $1.24
v2Question-type answer shapes, strict span extraction, relative-time and count rules0.365714 min · $1.90
v3Cached baseline atomic facts ranked ahead of raw dialogue0.336217 min · $2.39
v4Cached MiniLM embeddings add eight semantic blocks for broad and inference queries0.362119 min · $2.71
v5Whole routed 14k-word conversation for list, count and broad questions0.337925 min · $4.02
v6Answer JSON gains an evidence field and a verification/calculation field0.388927 min · $4.51
v7Ordered temporal policy: compute long offsets, keep last-week and weekday relations0.444829 min · $4.95
v8Relation expansions for nicknames, consoles, travel, careers; constraint hypothesescat-3, 11 QA: 0.131734 min · $6.11
v9Whole routed conversation appended for explicitly inferential wordingcat-3, 11 QA: 0.138736 min · $6.37
v11Travel expansion only for undated state questions, occupation-only answer shapecat-3, 11 QA: 0.302239 min · $7.19
v12One LLM global participant/timeline/aggregate profile per conversation, cachedRetrieval cannot count. Add a second memory layer, an LLM profile of participants, timeline and totals, for broad questions.conv-0 cat-1: 0.323540 min · $7.54
v13Six lossless five-session ledgers plus a global synthesis replace the profileconv-0 cat-1: 0.306644 min · $8.43
v14BM25-selected ledger bullets beside the synthesis, exact predicate match requiredfull conv-0: 0.534847 min · $9.64
v15Exact-predicate warning scoped to profiles; temporal questions bypass profiles0.432154 min · $11.82
v16Counts get relation-focused occurrence bullets only, names off the rankingconv-0 cat-1: 0.335857 min · $12.93
v17Occurrence-only context for action counts, synthesis for noun-inventory countsconv-1-2 all: 0.546558 min · $13.53
v18Occurrence-only mode widened to 'how many of' and selected participlescount subset: 0.615459 min · $13.95
v19Second independent audit call for inference, coreference and description triggersconv-1-2 cat-3: 0.241971 min · $18.58
v20Second call narrowed to game and console entity resolutionconsole/card QA only75 min · $20.36
v21Hidden-role social-deduction guidance for impostor board games1 target QA: 0.000076 min · $20.73
v22Query-focused lossless ledgers for dates, durations and time-sliced countsconv-1-2 cat-2: 0.487877 min · $21.25
v23Lossless profile chunks cut from five sessions to threeconv-1 cat-1: 0.271779 min · $22.36
v24Profile and ledgers placed before raw snippets for plural and count inventories0.436983 min · $24.12
v25Second completeness/precision audit call for plural list questionslist subset: 0.532387 min · $26.35
v26Marked stem indexed for every query and document tokentune cat-1: 0.549790 min · $27.37
v27Ordinary stems for inflected words only, exact surface tokens preserved0.458193 min · $27.76
v28Answerer copies the smallest complete evidence clause for why/how/advice questionsBottleneck is the answer stage, not retrieval: copy the smallest complete evidence clause instead of one keyword.0.460398 min · $28.52
v29Retrieval chunks widened to 8 turns stride 3, selected blocks cut to 11/140.4451107 min · $30.24
v30Profile-first context for plural noun phrases before has/have/does/do31 changed QA: 0.4610112 min · $31.45
v31Only unambiguous 'which of', 'which new', 'who or which' go profile-first4 changed QA: 0.1882114 min · $31.94
v32Lossless ISO date formatting and required duration units on how-many-X answersreplay delta: +0.0025115 min · $32.43
v33Second inference audit for underlying, hypothetical and geographic triggers10 target QA: 0.0667116 min · $32.79
v34Restore 'the weekday before session date' when the absolute date checks out0.5001120 min · $34.03
v35Personal and possessive pronouns removed from routed BM25 queries0.5163123 min · $34.92
v36Three nonduplicate MiniLM semantic hits appended after all lexical blocks0.4908129 min · $37.13
v37Selected six-turn lexical blocks cut from 13/18 to 10/140.4847131 min · $38.51
v38Evidence reranked by normalized BM25 plus unique query-term coverage0.5098134 min · $39.77
v39Inferential yes/no answers must carry exact evidence, direct ones stay terseconv-1-2 yes/no: 0.7374138 min · $41.38
v40Inferential polarity reasons must add a clue absent from the questionno repeatable gain139 min · $42.29
v41Nearest prior answer (Jaccard >= 0.80) prepended as a consistency hint70 changed QA: -0.0098143 min · $43.73
v42Query-focused occurrence ledgers for the 14 ordinal temporal questionsconv-0 ordinals: 0.4932148 min · $46.26
43 snapshots, $48.21, 159 min, priciest run. conv-50 held back as an untouched audit: 0.5609 there, 0.5716 sealed.
v1Inherited baseline: LLM extraction, BM25 k=5, 8-entry context, one answer call0.26951 min · $0.16
v2Rewrite: no LLM extraction, raw transcript windows, MiniLM+BM25, 14-seed contextDelete the extraction stage instead of improving it: index the raw transcript losslessly and let retrieval do the summarising.0.304415 min · $1.96
v3Question-type retrieval budgets (7 / 20-24 / 12 seeds) and relative-time wording0.350918 min · $2.36
v4Salutation-based nickname recovery, constraint inference, temporal templates0.363220 min · $2.72
v5Question-only LLM planner issuing two paraphrased searches, fused by RRF0.362126 min · $3.71
v6Aggressive cross-entity semantic transfer aimed at adversarial questionscat-5 only: 0.422732 min · $4.96
v7Dense/lexical fusion reweighted from 0.56/0.44 to 0.45/0.550.315937 min · $5.92
v8BGE-base embeddings in place of MiniLM0.354447 min · $8.13
v9Answer JSON with an evidence / conflict / relative-date check before the answerSame retrieval, different answering contract: force an evidence and conflict check in JSON, then return only the answer field.0.393252 min · $9.40
v10Twenty seeds for inference questions plus clue-and-world-knowledge instructioncat-3 only: 0.146460 min · $11.24
v12Temporal and factoid retrieval cap raised from 7 to 12 seeds0.42667 min · $13.23
v13Temporal and factoid cap raised again from 12 to 16 seedscat-2 only: 0.543370 min · $14.04
v14Aggregate and count budgets widened to 30 and 35 seedscat-1 only: 0.382476 min · $15.59
v15Aggregate and count budgets tightened to 14 and 20 seedscat-1 only: 0.375979 min · $16.38
v16Raw windows and inherited atomic facts merged into one joint index0.423596 min · $21.19
v17Three atomic-fact results appended after v12's unchanged raw ranking0.4318100 min · $22.41
v18Retry any first answer on the refusal floor with a forced-inference promptTreat 'unknown' as a retryable failure, not an answer: re-ask once with a prompt forcing a best evidence-grounded guess.0.4228106 min · $24.16
v19Namespace intersection for multi-participant questions, wider refusal detectioncontract test only108 min · $25.20
v20Parse JSON before regex so numeric and boolean answers become short stringsparser unit tests only109 min · $25.62
v21Conservative second-pass verifier over every non-refusal answer0.4217118 min · $28.35
v22Deterministic cleanup stripping trailing Evidence / context citationsparser unit tests only118 min · $28.55
22 snapshots, $28.64, 119 min, cheapest run: no LLM extraction, so ingest took 15 s not 9 min. Most scores are one 60-QA slice; the last four none.
v2Full rewrite: parallel per-session fact extraction, verbatim windows, BM25+bge RRFOne extraction call per session, dated facts kept beside verbatim dialogue windows, hybrid lexical+dense retrieval, never refuse.0.518325 min · $2.49
v3Answer prompt: topic-over-name fallback, never refuse, 30 facts + 8 windows0.528138 min · $3.45
v4Brevity-first rule list (1-5 words, third person) plus five style examplesGold answers are phrases, not sentences, so length and exact wording go first in the rules, with worked examples.0.547138 min · $3.45
v5Three whole sessions as context instead of eight dialogue windows0.5455$4.20
v6Two whole sessions plus five non-overlapping dialogue windows0.5496$4.96
v7Adds speaker profiles and a dated session timeline; only the timeline was kept0.556$5.71
v8Answer JSON forced to {quote, answer} so every answer is a verbatim span0.4882$6.46
v9Question-type TARGET FORM hints computed in code (yes/no, how many, when, list)Gold-length analysis showed lists under-answered and yes/no over-answered; the fix is a code-computed answer shape, not retrieval.0.5734$7.22
v10Entity-centric aggregation index for list and count questions0.574966 min · $7.97
No trajectory curve: this run left one comparable self-check measurement, so there is nothing to plot against spend. The versions and what each one changed are below.
v3Raw-dialogue hybrid retrieval, query expansion, temporal resolver, session notesDelete the ingest LLM: index raw dialogue at turn, window and session scale, and rewrite absolute dates into relative wording.conv-0 260 QA: 0.5281
v7Speaker profiles from notes, topic-turn dump for how-many, answer tighteningAnswer identity and counting questions from a profile built over the notes rather than from whatever the query lexically matches.0.5074
v8Same-month session boost, force-include when the question names a month or yearnot re-scored
5 snapshots, 99 min; this harness reports no token or cost data. Only v7 ran the full 812-QA self-check, with conv-50 held out until then.
No trajectory curve: this run left one comparable self-check measurement, so there is nothing to plot against spend. The versions and what each one changed are below.
v1MiniLM+BM25 hybrid over extracted memories, raw-turn backup index, dated answersKeep the inherited extractor and index raw turns beside it, then print every date as relative phrase plus absolute date.0.417360 min · $1.88
One snapshot, $1.96, 60 min, cheapest run. Everything was tried in main/ and versioned once at the end; the 0.2503 baseline was never snapshotted.