vrr/ruvector
2
0
Fork 0
mirror of https://github.com/ruvnet/RuVector.git synced 2026-08-07 15:53:50 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 15
ruvector/examples/mragent/data/eval-set.json
rUv edf96d83ed
feat(mragent): self-reconstructing graph memory over RuVector, evolved by Darwin (ADR-269/270) (#611) 
* feat(mragent): MRAgent graph memory over RuVector with Darwin optimization

Add ADR-269 and a runnable reference implementation of MRAgent ("Memory is
Reconstructed, Not Retrieved") on RuVector, optimized by Meta-Harness Darwin
Mode under the "freeze the model, evolve the harness" invariant.

- Frozen model: deterministic Cue-Tag-Content memory substrate mirroring
  RuVector hybrid (RRF) search + bounded-depth Cypher traversal semantics
  (examples/mragent/agent/memory.mjs)
- Evolved harness: 10-gene reconstruction genome (cueK, efSearch, hybridAlpha,
  fusion, traversalDepth, tagFanout, pruneThreshold, maxContent, rerank,
  promptStrategy) in DARWIN_MUTABLE_BLOCK regions (agent/harness.mjs)
- Darwin evolution loop with mapLimit/paretoFront and ADR-150 graceful fallback
  when @metaharness/darwin is absent (optimize.mjs)
- scorePolicy.ts fitness mirroring ADR-266; benchmark + probe + 7 deterministic
  acceptance gates
- eval corpus with chained multi-hop "bridge" tasks so traversal depth, fan-out
  and pruning are genuinely load-bearing

Runs with zero optional deps: baseline 83.3% -> evolved 100% accuracy, faster
and ~33% smaller context. Darwin discovers traversalDepth=3 (LINKED_TO*1..3).

Co-Authored-By: claude-flow <ruv@ruv.net>
Claude-Session: https://claude.ai/code/session_017MDmEV4svuFxuDBGg8zek2

* feat(mragent): self-reconstructing graph memory, beyond SOTA (ADR-270)

Extend the MRAgent harness past the paper into calibrated, adaptive,
self-reorganizing memory, co-evolved by Darwin. Also fixes the corpus being
silently excluded by the root .gitignore data/ rule (the example was missing
its eval set).

Beyond-SOTA mechanisms (each a tunable gene Darwin evolves):
- Adaptive depth (haltConfidence): halt traversal once evidence is decisive
- Abstention + risk-adjusted utility (abstainThreshold): refuse on weak
  evidence instead of hallucinating; graded on calibrated utility, not raw acc
- Consolidation/replay (agent/consolidate.mjs): store reorganizes its own
  topology, laying Cue->shortcut->Content edges (RuVector self-learning GNN)

Substrate upgrades:
- Concept layer (agent/concepts.mjs): dense (concept) vs sparse (token) signals
  genuinely decoupled, so hybridAlpha/fusion become load-bearing
- Hardened 24-task corpus, 6 classes (semantic/lexical/hybrid/bridge/
  distractor/unanswerable) synthesized from structured signal specs
- All 12 genes proven load-bearing (some via epistatic interaction)
- Memetic optimizer: GA (mapLimit/paretoFront) + multi-start coordinate-descent
  polish that reliably finds the narrow calibration optimum

Measured (deterministic, zero optional deps): baseline acc 81% / risk 0.708 /
halluc 0.13 -> evolved 100% / risk 1.000 / halluc 0.00; consolidation -25%
hops at 100% accuracy. 11 acceptance gates pass. ADR-150 compliant.

Co-Authored-By: claude-flow <ruv@ruv.net>
Claude-Session: https://claude.ai/code/session_017MDmEV4svuFxuDBGg8zek2

* feat(mragent): generalization protocol (train/test/CV) + overfit fixes

Add a held-out evaluation regime that proves the evolved harness GENERALIZES
rather than memorizing the eval set, and fix the overfitting it surfaced.

Protocol:
- Scale corpus to 60 tasks via a deterministic generator (tools/genCorpus.mjs,
  npm run gen-corpus), 10 per class, difficulty-varied (1-hop AND 2-hop bridges,
  1-3 ranking-distractors) so train constrains every gene
- Optimizer evolves on a class-stratified TRAIN split, selects via 3-fold
  cross-validation with a variance penalty (mean - 0.5*range), and reports a
  held-out TEST split it never saw
- Generalization gate = does evolution improve the unseen split

Overfit fixes uncovered by held-out eval:
- Abstention confidence now derives from the answer's RAW relevance, not its
  decay^depth path score, so deep-but-relevant bridge answers aren't mistaken
  for weak ones (b-test confidence 0.39 -> 0.79); abstention generalizes across
  depths. Adaptive-depth halt uses the same raw-relevance signal.
- Larger difficulty-varied corpus + CV variance penalty stop the optimizer
  shaving under-constrained genes (maxContent->1) to train-fragile settings

Result (held-out test, reproducible): baseline ~30% acc / risk 0.25 / halluc
0.17 -> evolved ~65% / 0.81 / 0.04 (+35pt acc, +0.56 risk). Honest ceiling
(~80%) documented: synthetic embedding noise + one global hybridAlpha can't
serve both dense- and sparse-keyed queries. 12 acceptance gates pass.

Co-Authored-By: claude-flow <ruv@ruv.net>
Claude-Session: https://claude.ai/code/session_017MDmEV4svuFxuDBGg8zek2

* feat(mragent): GPU LLM write-layer for the Darwin optimizer (local RTX 5080)

Adds the directed-proposal layer the GA lacks (ADR-260 'real Darwin write-layer
proposes leaps from failure traces'): agent/llmMutator.mjs shows a local,
GPU-served code model (qwen2.5-coder via an OpenAI-compatible endpoint) the
current genome + its failing cases and asks for improved genomes. Every proposal
is clamped to the declared gene bounds (coerceGenome) before entering the
population, so untrusted LLM output can only ever be a safe genome — never an
unsafe gene. Wired into optimize.mjs every 3rd generation; folded into the
archive so GPU candidates compete in polish + acceptance.

Fully opt-in + gracefully degrading (ADR-150): MRAGENT_LLM=off or no reachable
endpoint => identical deterministic GA+coordinate-descent run as before. Auto-
detects http://localhost:11434/v1 (ollama) by default; MRAGENT_LLM_URL/MODEL
override.

Measured (RTX 5080, qwen2.5-coder:7b): 8 genomes proposed across gens, bounds-
safe; the deterministic polish still wins on this small synthetic corpus (the
GA+grid already enumerates the optimum), so the write-layer is a no-regression
enhancement that matters on larger corpora the grid can't cover. 14/14 tests
pass (2 new coerceGenome safety tests).

Co-Authored-By: claude-flow <ruv@ruv.net>

---------

Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: ruvnet <ruvnet@gmail.com>
2026-06-27 11:08:26 -04:00

1685 lines
28 KiB
JSON
Raw Permalink Blame History

This file contains ambiguous Unicode characters

This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

{
"_comment": "GENERATED by tools/genCorpus.mjs — do not hand-edit. Class-balanced, difficulty-varied structured signal specs; agent/memory.mjs synthesizes Cue/Tag/Content node texts. 6 classes × 10 = 60 tasks. Bridges alternate 1-hop/2-hop; distractors vary 1–3. Regenerate: node tools/genCorpus.mjs",
"tasks": [
{
"id": "s0",
"class": "semantic",
"prompt": "semantic 0",
"expected_fact": "sfact0",
"qConcepts": [
"fast",
"store"
],
"qLex": [
"s0tok"
],
"cue": {
"concepts": [
"fast",
"store"
],
"lex": []
},
"decoys": [
{
"concepts": [],
"lex": [
"s0tok"
]
}
]
},
{
"id": "l0",
"class": "lexical",
"prompt": "lexical 0",
"expected_fact": "lfact0",
"qConcepts": [
"boot"
],
"qLex": [
"lx-0"
],
"cue": {
"concepts": [],
"lex": [
"lx-0"
]
},
"decoys": [
{
"concepts": [
"boot"
],
"lex": []
}
]
},
{
"id": "h0",
"class": "hybrid",
"prompt": "hybrid 0",
"expected_fact": "hfact0",
"qConcepts": [
"fast",
"graph"
],
"qLex": [
"hy-0"
],
"cue": {
"concepts": [
"fast"
],
"lex": [
"hy-0"
]
},
"decoys": [
{
"concepts": [
"fast",
"graph"
],
"lex": []
},
{
"concepts": [],
"lex": [
"hy-0"
]
}
]
},
{
"id": "b0",
"class": "bridge",
"prompt": "bridge 0",
"expected_fact": "bfact0",
"qConcepts": [
"compress",
"secure"
],
"qLex": [
"b0tok"
],
"cue": {
"concepts": [
"compress",
"secure"
],
"lex": [
"b0tok"
]
},
"bridges": 1
},
{
"id": "d0",
"class": "distractor",
"prompt": "distractor 0",
"expected_fact": "dfact0",
"qConcepts": [
"search",
"prune"
],
"qLex": [
"d0tok"
],
"cue": {
"concepts": [
"search",
"prune"
],
"lex": [
"d0tok"
]
},
"distractors": 1,
"corroborate": true
},
{
"id": "u0",
"class": "unanswerable",
"answerable": false,
"prompt": "unanswerable 0",
"expected_fact": "N/A",
"qConcepts": [
"weather",
"ballet"
],
"qLex": [
"u0tok"
],
"cue": {
"concepts": [
"weather",
"ballet"
],
"lex": [
"u0tok"
]
},
"decoys": [
{
"concepts": [],
"lex": [
"u0tok"
]
}
]
},
{
"id": "s1",
"class": "semantic",
"prompt": "semantic 1",
"expected_fact": "sfact1",
"qConcepts": [
"boot",
"search"
],
"qLex": [
"s1tok"
],
"cue": {
"concepts": [
"boot",
"search"
],
"lex": []
},
"decoys": [
{
"concepts": [],
"lex": [
"s1tok"
]
}
]
},
{
"id": "l1",
"class": "lexical",
"prompt": "lexical 1",
"expected_fact": "lfact1",
"qConcepts": [
"compress"
],
"qLex": [
"lx-1"
],
"cue": {
"concepts": [],
"lex": [
"lx-1"
]
},
"decoys": [
{
"concepts": [
"compress"
],
"lex": []
}
]
},
{
"id": "h1",
"class": "hybrid",
"prompt": "hybrid 1",
"expected_fact": "hfact1",
"qConcepts": [
"boot",
"consensus"
],
"qLex": [
"hy-1"
],
"cue": {
"concepts": [
"boot"
],
"lex": [
"hy-1"
]
},
"decoys": [
{
"concepts": [
"boot",
"consensus"
],
"lex": []
},
{
"concepts": [],
"lex": [
"hy-1"
]
}
]
},
{
"id": "b1",
"class": "bridge",
"prompt": "bridge 1",
"expected_fact": "bfact1",
"qConcepts": [
"store",
"merge"
],
"qLex": [
"b1tok"
],
"cue": {
"concepts": [
"store",
"merge"
],
"lex": [
"b1tok"
]
},
"bridges": 2
},
{
"id": "d1",
"class": "distractor",
"prompt": "distractor 1",
"expected_fact": "dfact1",
"qConcepts": [
"graph",
"accuracy"
],
"qLex": [
"d1tok"
],
"cue": {
"concepts": [
"graph",
"accuracy"
],
"lex": [
"d1tok"
]
},
"distractors": 2,
"corroborate": true
},
{
"id": "u1",
"class": "unanswerable",
"answerable": false,
"prompt": "unanswerable 1",
"expected_fact": "N/A",
"qConcepts": [
"tariff",
"comet"
],
"qLex": [
"u1tok"
],
"cue": {
"concepts": [
"tariff",
"comet"
],
"lex": [
"u1tok"
]
},
"decoys": [
{
"concepts": [],
"lex": [
"u1tok"
]
}
]
},
{
"id": "s2",
"class": "semantic",
"prompt": "semantic 2",
"expected_fact": "sfact2",
"qConcepts": [
"compress",
"graph"
],
"qLex": [
"s2tok"
],
"cue": {
"concepts": [
"compress",
"graph"
],
"lex": []
},
"decoys": [
{
"concepts": [],
"lex": [
"s2tok"
]
}
]
},
{
"id": "l2",
"class": "lexical",
"prompt": "lexical 2",
"expected_fact": "lfact2",
"qConcepts": [
"store"
],
"qLex": [
"lx-2"
],
"cue": {
"concepts": [],
"lex": [
"lx-2"
]
},
"decoys": [
{
"concepts": [
"store"
],
"lex": []
}
]
},
{
"id": "h2",
"class": "hybrid",
"prompt": "hybrid 2",
"expected_fact": "hfact2",
"qConcepts": [
"compress",
"secure"
],
"qLex": [
"hy-2"
],
"cue": {
"concepts": [
"compress"
],
"lex": [
"hy-2"
]
},
"decoys": [
{
"concepts": [
"compress",
"secure"
],
"lex": []
},
{
"concepts": [],
"lex": [
"hy-2"
]
}
]
},
{
"id": "b2",
"class": "bridge",
"prompt": "bridge 2",
"expected_fact": "bfact2",
"qConcepts": [
"search",
"prune"
],
"qLex": [
"b2tok"
],
"cue": {
"concepts": [
"search",
"prune"
],
"lex": [
"b2tok"
]
},
"bridges": 1
},
{
"id": "d2",
"class": "distractor",
"prompt": "distractor 2",
"expected_fact": "dfact2",
"qConcepts": [
"consensus",
"memory"
],
"qLex": [
"d2tok"
],
"cue": {
"concepts": [
"consensus",
"memory"
],
"lex": [
"d2tok"
]
},
"distractors": 3,
"corroborate": true
},
{
"id": "u2",
"class": "unanswerable",
"answerable": false,
"prompt": "unanswerable 2",
"expected_fact": "N/A",
"qConcepts": [
"opera",
"cuisine"
],
"qLex": [
"u2tok"
],
"cue": {
"concepts": [
"opera",
"cuisine"
],
"lex": [
"u2tok"
]
},
"decoys": [
{
"concepts": [],
"lex": [
"u2tok"
]
}
]
},
{
"id": "s3",
"class": "semantic",
"prompt": "semantic 3",
"expected_fact": "sfact3",
"qConcepts": [
"store",
"consensus"
],
"qLex": [
"s3tok"
],
"cue": {
"concepts": [
"store",
"consensus"
],
"lex": []
},
"decoys": [
{
"concepts": [],
"lex": [
"s3tok"
]
}
]
},
{
"id": "l3",
"class": "lexical",
"prompt": "lexical 3",
"expected_fact": "lfact3",
"qConcepts": [
"search"
],
"qLex": [
"lx-3"
],
"cue": {
"concepts": [],
"lex": [
"lx-3"
]
},
"decoys": [
{
"concepts": [
"search"
],
"lex": []
}
]
},
{
"id": "h3",
"class": "hybrid",
"prompt": "hybrid 3",
"expected_fact": "hfact3",
"qConcepts": [
"store",
"merge"
],
"qLex": [
"hy-3"
],
"cue": {
"concepts": [
"store"
],
"lex": [
"hy-3"
]
},
"decoys": [
{
"concepts": [
"store",
"merge"
],
"lex": []
},
{
"concepts": [],
"lex": [
"hy-3"
]
}
]
},
{
"id": "b3",
"class": "bridge",
"prompt": "bridge 3",
"expected_fact": "bfact3",
"qConcepts": [
"graph",
"accuracy"
],
"qLex": [
"b3tok"
],
"cue": {
"concepts": [
"graph",
"accuracy"
],
"lex": [
"b3tok"
]
},
"bridges": 2
},
{
"id": "d3",
"class": "distractor",
"prompt": "distractor 3",
"expected_fact": "dfact3",
"qConcepts": [
"secure",
"validate"
],
"qLex": [
"d3tok"
],
"cue": {
"concepts": [
"secure",
"validate"
],
"lex": [
"d3tok"
]
},
"distractors": 1,
"corroborate": true
},
{
"id": "u3",
"class": "unanswerable",
"answerable": false,
"prompt": "unanswerable 3",
"expected_fact": "N/A",
"qConcepts": [
"sprint",
"monsoon"
],
"qLex": [
"u3tok"
],
"cue": {
"concepts": [
"sprint",
"monsoon"
],
"lex": [
"u3tok"
]
},
"decoys": [
{
"concepts": [],
"lex": [
"u3tok"
]
}
]
},
{
"id": "s4",
"class": "semantic",
"prompt": "semantic 4",
"expected_fact": "sfact4",
"qConcepts": [
"search",
"secure"
],
"qLex": [
"s4tok"
],
"cue": {
"concepts": [
"search",
"secure"
],
"lex": []
},
"decoys": [
{
"concepts": [],
"lex": [
"s4tok"
]
}
]
},
{
"id": "l4",
"class": "lexical",
"prompt": "lexical 4",
"expected_fact": "lfact4",
"qConcepts": [
"graph"
],
"qLex": [
"lx-4"
],
"cue": {
"concepts": [],
"lex": [
"lx-4"
]
},
"decoys": [
{
"concepts": [
"graph"
],
"lex": []
}
]
},
{
"id": "h4",
"class": "hybrid",
"prompt": "hybrid 4",
"expected_fact": "hfact4",
"qConcepts": [
"search",
"prune"
],
"qLex": [
"hy-4"
],
"cue": {
"concepts": [
"search"
],
"lex": [
"hy-4"
]
},
"decoys": [
{
"concepts": [
"search",
"prune"
],
"lex": []
},
{
"concepts": [],
"lex": [
"hy-4"
]
}
]
},
{
"id": "b4",
"class": "bridge",
"prompt": "bridge 4",
"expected_fact": "bfact4",
"qConcepts": [
"consensus",
"memory"
],
"qLex": [
"b4tok"
],
"cue": {
"concepts": [
"consensus",
"memory"
],
"lex": [
"b4tok"
]
},
"bridges": 1
},
{
"id": "d4",
"class": "distractor",
"prompt": "distractor 4",
"expected_fact": "dfact4",
"qConcepts": [
"merge",
"concurrency"
],
"qLex": [
"d4tok"
],
"cue": {
"concepts": [
"merge",
"concurrency"
],
"lex": [
"d4tok"
]
},
"distractors": 2,
"corroborate": true
},
{
"id": "u4",
"class": "unanswerable",
"answerable": false,
"prompt": "unanswerable 4",
"expected_fact": "N/A",
"qConcepts": [
"recipe",
"harpsichord"
],
"qLex": [
"u4tok"
],
"cue": {
"concepts": [
"recipe",
"harpsichord"
],
"lex": [
"u4tok"
]
},
"decoys": [
{
"concepts": [],
"lex": [
"u4tok"
]
}
]
},
{
"id": "s5",
"class": "semantic",
"prompt": "semantic 5",
"expected_fact": "sfact5",
"qConcepts": [
"graph",
"merge"
],
"qLex": [
"s5tok"
],
"cue": {
"concepts": [
"graph",
"merge"
],
"lex": []
},
"decoys": [
{
"concepts": [],
"lex": [
"s5tok"
]
}
]
},
{
"id": "l5",
"class": "lexical",
"prompt": "lexical 5",
"expected_fact": "lfact5",
"qConcepts": [
"consensus"
],
"qLex": [
"lx-5"
],
"cue": {
"concepts": [],
"lex": [
"lx-5"
]
},
"decoys": [
{
"concepts": [
"consensus"
],
"lex": []
}
]
},
{
"id": "h5",
"class": "hybrid",
"prompt": "hybrid 5",
"expected_fact": "hfact5",
"qConcepts": [
"graph",
"accuracy"
],
"qLex": [
"hy-5"
],
"cue": {
"concepts": [
"graph"
],
"lex": [
"hy-5"
]
},
"decoys": [
{
"concepts": [
"graph",
"accuracy"
],
"lex": []
},
{
"concepts": [],
"lex": [
"hy-5"
]
}
]
},
{
"id": "b5",
"class": "bridge",
"prompt": "bridge 5",
"expected_fact": "bfact5",
"qConcepts": [
"secure",
"validate"
],
"qLex": [
"b5tok"
],
"cue": {
"concepts": [
"secure",
"validate"
],
"lex": [
"b5tok"
]
},
"bridges": 2
},
{
"id": "d5",
"class": "distractor",
"prompt": "distractor 5",
"expected_fact": "dfact5",
"qConcepts": [
"prune",
"embedding"
],
"qLex": [
"d5tok"
],
"cue": {
"concepts": [
"prune",
"embedding"
],
"lex": [
"d5tok"
]
},
"distractors": 3,
"corroborate": true
},
{
"id": "u5",
"class": "unanswerable",
"answerable": false,
"prompt": "unanswerable 5",
"expected_fact": "N/A",
"qConcepts": [
"planet",
"tundra"
],
"qLex": [
"u5tok"
],
"cue": {
"concepts": [
"planet",
"tundra"
],
"lex": [
"u5tok"
]
},
"decoys": [
{
"concepts": [],
"lex": [
"u5tok"
]
}
]
},
{
"id": "s6",
"class": "semantic",
"prompt": "semantic 6",
"expected_fact": "sfact6",
"qConcepts": [
"consensus",
"prune"
],
"qLex": [
"s6tok"
],
"cue": {
"concepts": [
"consensus",
"prune"
],
"lex": []
},
"decoys": [
{
"concepts": [],
"lex": [
"s6tok"
]
}
]
},
{
"id": "l6",
"class": "lexical",
"prompt": "lexical 6",
"expected_fact": "lfact6",
"qConcepts": [
"secure"
],
"qLex": [
"lx-6"
],
"cue": {
"concepts": [],
"lex": [
"lx-6"
]
},
"decoys": [
{
"concepts": [
"secure"
],
"lex": []
}
]
},
{
"id": "h6",
"class": "hybrid",
"prompt": "hybrid 6",
"expected_fact": "hfact6",
"qConcepts": [
"consensus",
"memory"
],
"qLex": [
"hy-6"
],
"cue": {
"concepts": [
"consensus"
],
"lex": [
"hy-6"
]
},
"decoys": [
{
"concepts": [
"consensus",
"memory"
],
"lex": []
},
{
"concepts": [],
"lex": [
"hy-6"
]
}
]
},
{
"id": "b6",
"class": "bridge",
"prompt": "bridge 6",
"expected_fact": "bfact6",
"qConcepts": [
"merge",
"concurrency"
],
"qLex": [
"b6tok"
],
"cue": {
"concepts": [
"merge",
"concurrency"
],
"lex": [
"b6tok"
]
},
"bridges": 1
},
{
"id": "d6",
"class": "distractor",
"prompt": "distractor 6",
"expected_fact": "dfact6",
"qConcepts": [
"accuracy",
"fast"
],
"qLex": [
"d6tok"
],
"cue": {
"concepts": [
"accuracy",
"fast"
],
"lex": [
"d6tok"
]
},
"distractors": 1,
"corroborate": true
},
{
"id": "u6",
"class": "unanswerable",
"answerable": false,
"prompt": "unanswerable 6",
"expected_fact": "N/A",
"qConcepts": [
"ballet",
"weather"
],
"qLex": [
"u6tok"
],
"cue": {
"concepts": [
"ballet",
"weather"
],
"lex": [
"u6tok"
]
},
"decoys": [
{
"concepts": [],
"lex": [
"u6tok"
]
}
]
},
{
"id": "s7",
"class": "semantic",
"prompt": "semantic 7",
"expected_fact": "sfact7",
"qConcepts": [
"secure",
"accuracy"
],
"qLex": [
"s7tok"
],
"cue": {
"concepts": [
"secure",
"accuracy"
],
"lex": []
},
"decoys": [
{
"concepts": [],
"lex": [
"s7tok"
]
}
]
},
{
"id": "l7",
"class": "lexical",
"prompt": "lexical 7",
"expected_fact": "lfact7",
"qConcepts": [
"merge"
],
"qLex": [
"lx-7"
],
"cue": {
"concepts": [],
"lex": [
"lx-7"
]
},
"decoys": [
{
"concepts": [
"merge"
],
"lex": []
}
]
},
{
"id": "h7",
"class": "hybrid",
"prompt": "hybrid 7",
"expected_fact": "hfact7",
"qConcepts": [
"secure",
"validate"
],
"qLex": [
"hy-7"
],
"cue": {
"concepts": [
"secure"
],
"lex": [
"hy-7"
]
},
"decoys": [
{
"concepts": [
"secure",
"validate"
],
"lex": []
},
{
"concepts": [],
"lex": [
"hy-7"
]
}
]
},
{
"id": "b7",
"class": "bridge",
"prompt": "bridge 7",
"expected_fact": "bfact7",
"qConcepts": [
"prune",
"embedding"
],
"qLex": [
"b7tok"
],
"cue": {
"concepts": [
"prune",
"embedding"
],
"lex": [
"b7tok"
]
},
"bridges": 2
},
{
"id": "d7",
"class": "distractor",
"prompt": "distractor 7",
"expected_fact": "dfact7",
"qConcepts": [
"memory",
"boot"
],
"qLex": [
"d7tok"
],
"cue": {
"concepts": [
"memory",
"boot"
],
"lex": [
"d7tok"
]
},
"distractors": 2,
"corroborate": true
},
{
"id": "u7",
"class": "unanswerable",
"answerable": false,
"prompt": "unanswerable 7",
"expected_fact": "N/A",
"qConcepts": [
"comet",
"tariff"
],
"qLex": [
"u7tok"
],
"cue": {
"concepts": [
"comet",
"tariff"
],
"lex": [
"u7tok"
]
},
"decoys": [
{
"concepts": [],
"lex": [
"u7tok"
]
}
]
},
{
"id": "s8",
"class": "semantic",
"prompt": "semantic 8",
"expected_fact": "sfact8",
"qConcepts": [
"merge",
"memory"
],
"qLex": [
"s8tok"
],
"cue": {
"concepts": [
"merge",
"memory"
],
"lex": []
},
"decoys": [
{
"concepts": [],
"lex": [
"s8tok"
]
}
]
},
{
"id": "l8",
"class": "lexical",
"prompt": "lexical 8",
"expected_fact": "lfact8",
"qConcepts": [
"prune"
],
"qLex": [
"lx-8"
],
"cue": {
"concepts": [],
"lex": [
"lx-8"
]
},
"decoys": [
{
"concepts": [
"prune"
],
"lex": []
}
]
},
{
"id": "h8",
"class": "hybrid",
"prompt": "hybrid 8",
"expected_fact": "hfact8",
"qConcepts": [
"merge",
"concurrency"
],
"qLex": [
"hy-8"
],
"cue": {
"concepts": [
"merge"
],
"lex": [
"hy-8"
]
},
"decoys": [
{
"concepts": [
"merge",
"concurrency"
],
"lex": []
},
{
"concepts": [],
"lex": [
"hy-8"
]
}
]
},
{
"id": "b8",
"class": "bridge",
"prompt": "bridge 8",
"expected_fact": "bfact8",
"qConcepts": [
"accuracy",
"fast"
],
"qLex": [
"b8tok"
],
"cue": {
"concepts": [
"accuracy",
"fast"
],
"lex": [
"b8tok"
]
},
"bridges": 1
},
{
"id": "d8",
"class": "distractor",
"prompt": "distractor 8",
"expected_fact": "dfact8",
"qConcepts": [
"validate",
"compress"
],
"qLex": [
"d8tok"
],
"cue": {
"concepts": [
"validate",
"compress"
],
"lex": [
"d8tok"
]
},
"distractors": 3,
"corroborate": true
},
{
"id": "u8",
"class": "unanswerable",
"answerable": false,
"prompt": "unanswerable 8",
"expected_fact": "N/A",
"qConcepts": [
"cuisine",
"opera"
],
"qLex": [
"u8tok"
],
"cue": {
"concepts": [
"cuisine",
"opera"
],
"lex": [
"u8tok"
]
},
"decoys": [
{
"concepts": [],
"lex": [
"u8tok"
]
}
]
},
{
"id": "s9",
"class": "semantic",
"prompt": "semantic 9",
"expected_fact": "sfact9",
"qConcepts": [
"prune",
"validate"
],
"qLex": [
"s9tok"
],
"cue": {
"concepts": [
"prune",
"validate"
],
"lex": []
},
"decoys": [
{
"concepts": [],
"lex": [
"s9tok"
]
}
]
},
{
"id": "l9",
"class": "lexical",
"prompt": "lexical 9",
"expected_fact": "lfact9",
"qConcepts": [
"accuracy"
],
"qLex": [
"lx-9"
],
"cue": {
"concepts": [],
"lex": [
"lx-9"
]
},
"decoys": [
{
"concepts": [
"accuracy"
],
"lex": []
}
]
},
{
"id": "h9",
"class": "hybrid",
"prompt": "hybrid 9",
"expected_fact": "hfact9",
"qConcepts": [
"prune",
"embedding"
],
"qLex": [
"hy-9"
],
"cue": {
"concepts": [
"prune"
],
"lex": [
"hy-9"
]
},
"decoys": [
{
"concepts": [
"prune",
"embedding"
],
"lex": []
},
{
"concepts": [],
"lex": [
"hy-9"
]
}
]
},
{
"id": "b9",
"class": "bridge",
"prompt": "bridge 9",
"expected_fact": "bfact9",
"qConcepts": [
"memory",
"boot"
],
"qLex": [
"b9tok"
],
"cue": {
"concepts": [
"memory",
"boot"
],
"lex": [
"b9tok"
]
},
"bridges": 2
},
{
"id": "d9",
"class": "distractor",
"prompt": "distractor 9",
"expected_fact": "dfact9",
"qConcepts": [
"concurrency",
"store"
],
"qLex": [
"d9tok"
],
"cue": {
"concepts": [
"concurrency",
"store"
],
"lex": [
"d9tok"
]
},
"distractors": 1,
"corroborate": true
},
{
"id": "u9",
"class": "unanswerable",
"answerable": false,
"prompt": "unanswerable 9",
"expected_fact": "N/A",
"qConcepts": [
"monsoon",
"sprint"
],
"qLex": [
"u9tok"
],
"cue": {
"concepts": [
"monsoon",
"sprint"
],
"lex": [
"u9tok"
]
},
"decoys": [
{
"concepts": [],
"lex": [
"u9tok"
]
}
]
}
]
}
Reference in a new issue View git blame Copy permalink