---
title: "Building a Benchmark"
description: "Add a custom benchmark dataset when the built-in ones don't match your use case"
sidebarTitle: "Building a Benchmark"
icon: "flask-conical"
---

MemoryBench ships with three datasets — [LoCoMo](https://github.com/snap-research/locomo), [LongMemEval](https://huggingface.co/datasets/xiaowu0162/longmemeval-cleaned), and [ConvoMem](https://huggingface.co/datasets/Salesforce/ConvoMem) — but they won't cover every product. If your agent has a workflow the built-in benchmarks don't exercise (a specific domain, a scenario your users actually hit, a regression you want to guard against), you can add your own benchmark dataset and every provider — including Supermemory — runs against it the same way.

## Choosing between the built-in datasets

Before building your own, check whether one of the existing datasets already covers what you need:

| Benchmark | Best for | Primary challenge |
|---|---|---|
| **LoCoMo** | Personal assistants, support bots — anything with recurring sessions over days/weeks | Temporal context, cross-session recall |
| **LongMemEval** | RAG / document search, knowledge bases | Information density, precise retrieval, synthesis |
| **ConvoMem** | Dialogue systems, interview bots, meeting assistants | Reference resolution within a single conversation |

You can also run all three to get a full picture of where a memory system is strong or weak — a system that's great at LoCoMo but weak at LongMemEval is good at temporal recall but struggles with dense information retrieval, for example.

## The Benchmark interface

Every dataset — built-in or custom — implements the same interface (`src/types/benchmark.ts`):

```typescript
interface Benchmark {
  name: string
  load(config?: BenchmarkConfig): Promise<void>
  getQuestions(filter?: QuestionFilter): UnifiedQuestion[]
  getHaystackSessions(questionId: string): UnifiedSession[]
  getGroundTruth(questionId: string): string
  getQuestionTypes(): QuestionTypeRegistry
}
```

Sessions — the raw conversational data a provider ingests before being asked a question — are normalized to a single shape regardless of benchmark:

```typescript
interface UnifiedSession {
  sessionId: string
  messages: Array<{ role: "user" | "assistant"; content: string }>
  metadata?: {
    date?: string           // ISO format
    formattedDate?: string  // human readable
    [key: string]: any
  }
}
```

## Adding your own benchmark

1. Create `src/benchmarks/mybenchmark/index.ts` implementing the `Benchmark` interface above
2. `load()` parses your dataset (JSON, CSV, whatever you have) into `UnifiedSession`s and questions
3. `getQuestions()` returns your question set, each with a ground-truth answer and a question type
4. Register it in `src/benchmarks/index.ts` and add the name to `BenchmarkName` in `src/types/benchmark.ts`
5. Run it exactly like a built-in benchmark:

```bash
bun run src/index.ts run -p supermemory -b mybenchmark
bun run src/index.ts compare -p supermemory,mem0,zep -b mybenchmark
```

Question types are your own vocabulary — group questions however matters to your product (`billing_history`, `escalation_context`, `preference_drift`, etc.), and the final report breaks accuracy down per type so you can see exactly where a provider is strong or weak on *your* scenarios, not a generic academic one.

<Note>
Full interface details and a working template: [`src/benchmarks/README.md`](https://github.com/supermemoryai/memorybench/blob/main/src/benchmarks/README.md) in the repo.
</Note>

---

## Next

<CardGroup cols={2}>
  <Card title="Adding a Provider" icon="plug" href="/memorybench/extend-provider">
    Register a memory system to run against your new benchmark.
  </Card>
  <Card title="Measuring Results" icon="gauge" href="/memorybench/memscore">
    How accuracy, latency, and MemScore are computed.
  </Card>
</CardGroup>