> This page is for version v2026-02-09 (default).
> For other versions, use one of these documentation indexes:
> - v2026-02-09 (default): https://docs.extend.ai/2026-02-09/llms.txt
> - v2025-04-21: https://docs.extend.ai/2025-04-21/llms.txt
> - v2024-12-23: https://docs.extend.ai/2024-12-23/llms.txt

> ## Documentation Index
> Fetch the complete documentation index at: https://docs.extend.ai/llms.txt
> Use this file to discover all available pages before exploring further.
>
> ## API version
> The current API version is `2026-02-09`, served at the site root (no version prefix in URLs).
> If this page URL contains `/2025-04-21/` or `/2024-12-23/`, you are reading an older API version.
> Prefer the current docs at https://docs.extend.ai/llms.txt unless the user explicitly needs that older version.
> Do not treat older-version pages as the source of truth for new integrations.

# Creating Evaluation Sets

> Create an evaluation set, upload files, and add items with expected outputs using the SDK (Python, TypeScript, Java, Go) or Extend Studio. Covers evaluationSets.create, files.upload, and evaluationSetItems.create.

An evaluation set is a collection of **items**: each item pairs a file (or, for extractors, a [package of files](#add-a-multifile-item)) with the output you expect a processor to produce for it. You create the set once against a primary extractor, classifier, or splitter, add items over time, and then [run the set](/2026-02-09/evaluation/running-evaluation-sets) to measure accuracy.

You can build a set entirely through the API, entirely in Extend Studio, or mix the two (for example, create items via the API and review them in Studio). Evaluation sets can also be run against workflows; this page covers sets for processors.

## Via the API

The full flow is four calls: create the set, upload a file, produce the expected output, and add the item. Use this when you want an evaluation loop in code, for example to rebuild a set from a folder of reviewed documents.

### Create the evaluation set

`entityId` is the primary processor the set is for. You can later run the set against any compatible processor, but every set has one default.

#### Python

```python
from extend_ai import Extend

client = Extend()

eval_set = client.evaluation_sets.create(
    name="Invoice extraction — Q3 vendors",
    description="Reviewed vendor invoices used to gate extractor releases",
    entity_id="ex_Xj8mK2pL9nR4vT7qY5wZ",
)
print(eval_set.id)  # "ev_..."
```

#### TypeScript

```typescript
import { ExtendClient } from "extend-ai";

const client = new ExtendClient();

const evalSet = await client.evaluationSets.create({
  name: "Invoice extraction — Q3 vendors",
  description: "Reviewed vendor invoices used to gate extractor releases",
  entityId: "ex_Xj8mK2pL9nR4vT7qY5wZ",
});
console.log(evalSet.id); // "ev_..."
```

#### Java

```java
import ai.extend.ExtendClient;
import ai.extend.resources.evaluationsets.requests.EvaluationSetsCreateRequest;
import ai.extend.types.EvaluationSet;

ExtendClient client = ExtendClient.builder().build();

EvaluationSet evalSet = client.evaluationSets().create(
    EvaluationSetsCreateRequest.builder()
        .name("Invoice extraction — Q3 vendors")
        .entityId("ex_Xj8mK2pL9nR4vT7qY5wZ")
        .description("Reviewed vendor invoices used to gate extractor releases")
        .build());
System.out.println(evalSet.getId()); // "ev_..."
```

#### Go

```go
package main

import (
	"context"
	"fmt"
	"log"

	extend "github.com/extend-hq/extend-go-sdk"
	client "github.com/extend-hq/extend-go-sdk/client"
)

func main() {
	c := client.NewClient()
	ctx := context.Background()

	evalSet, err := c.EvaluationSets.Create(ctx, &extend.EvaluationSetsCreateRequest{
		Name:        "Invoice extraction — Q3 vendors",
		Description: extend.String("Reviewed vendor invoices used to gate extractor releases"),
		EntityID:    "ex_Xj8mK2pL9nR4vT7qY5wZ",
	})
	if err != nil {
		log.Fatal(err)
	}
	fmt.Println(evalSet.ID) // "ev_..."
}
```

### Upload the file

Items reference files by Extend file ID (`file_...`). Upload each document once; you can reuse the same file ID in multiple evaluation sets.

#### Python

```python
with open("invoices/acme-2026-07.pdf", "rb") as f:
    uploaded = client.files.upload(file=f)
print(uploaded.id)  # "file_..."
```

#### TypeScript

```typescript
import { createReadStream } from "fs";

const uploaded = await client.files.upload(createReadStream("invoices/acme-2026-07.pdf"), {});
console.log(uploaded.id); // "file_..."
```

#### Java

```java
import ai.extend.resources.files.requests.FilesUploadRequest;
import ai.extend.types.File;

File uploaded = client.files().upload(
    new java.io.File("invoices/acme-2026-07.pdf"),
    FilesUploadRequest.builder().build());
System.out.println(uploaded.getId()); // "file_..."
```

#### Go

```go
// Requires "os" in your imports.
f, err := os.Open("invoices/acme-2026-07.pdf")
if err != nil {
	log.Fatal(err)
}
defer f.Close()

uploaded, err := c.Files.Upload(ctx, f, &extend.FilesUploadRequest{})
if err != nil {
	log.Fatal(err)
}
fmt.Println(uploaded.ID) // "file_..."
```

### Produce the expected output

The expected output is the ground truth the run is scored against. If you already have reviewed ground truth for the document (from an existing system, a spreadsheet, or a human review), map it to the shape below and use it directly; this is the preferred source because it is independent of the processor you are evaluating. The shape depends on the processor type:

| Processor  | `expectedOutput` shape                                                                                                                        |
| ---------- | --------------------------------------------------------------------------------------------------------------------------------------------- |
| Extractor  | `{ "value": { ...fields conforming to the extractor's schema... } }` — nest the object under `value` and include every property in the schema |
| Classifier | `{ "id": "<classification id>", "type": "<classification name>" }`                                                                            |
| Splitter   | `{ "splits": [{ "classificationId", "type", "startPage", "endPage" }, ...] }`                                                                 |

If you do not have ground truth yet, bootstrap it from a run: process the file, correct any mistakes, and save the corrected result. For an extractor, the run's output already matches the schema's field names and types:

#### Python

```python
run = client.extract(
    file={"id": uploaded.id},
    extractor={"id": "ex_Xj8mK2pL9nR4vT7qY5wZ"},
)

# Correct anything the extractor got wrong before saving it as ground truth
expected_value = dict(run.output.value)
expected_value["total"] = {"amount": 1249.00, "iso_4217_currency_code": "USD"}
```

#### TypeScript

```typescript
import { Extend } from "extend-ai";

const run = await client.extract({
  file: { id: uploaded.id },
  extractor: { id: "ex_Xj8mK2pL9nR4vT7qY5wZ" },
});
const output = run.output as Extend.ExtractOutputJson;

// Correct anything the extractor got wrong before saving it as ground truth
const expectedValue = {
  ...output.value,
  total: { amount: 1249.0, iso_4217_currency_code: "USD" },
};
```

#### Java

```java
import ai.extend.requests.ExtractRequest;
import ai.extend.types.ExtractOutputJson;
import ai.extend.types.ExtractRequestExtractor;
import ai.extend.types.ExtractRequestFile;
import ai.extend.types.ExtractRun;
import ai.extend.types.FileFromId;
import java.util.HashMap;
import java.util.Map;

ExtractRun run = client.extract(ExtractRequest.builder()
    .file(ExtractRequestFile.of(FileFromId.builder().id(uploaded.getId()).build()))
    .extractor(ExtractRequestExtractor.builder().id("ex_Xj8mK2pL9nR4vT7qY5wZ").build())
    .build());

ExtractOutputJson output = (ExtractOutputJson) run.getOutput().get().get();

// Correct anything the extractor got wrong before saving it as ground truth
Map<String, Object> expectedValue = new HashMap<>(output.getValue());
expectedValue.put("total", Map.of("amount", 1249.00, "iso_4217_currency_code", "USD"));
```

#### Go

```go
run, err := c.Extract(ctx, &extend.ExtractRequest{
	File:      &extend.ExtractRequestFile{FileFromID: &extend.FileFromID{ID: uploaded.ID}},
	Extractor: &extend.ExtractRequestExtractor{ID: "ex_Xj8mK2pL9nR4vT7qY5wZ"},
})
if err != nil {
	log.Fatal(err)
}

// Correct anything the extractor got wrong before saving it as ground truth
expectedValue := map[string]any{}
for k, v := range run.Output.GetExtractOutputJSON().Value {
	expectedValue[k] = v
}
expectedValue["total"] = map[string]any{"amount": 1249.00, "iso_4217_currency_code": "USD"}
```

### Add the item

You can add up to 100 items in one call. Each item needs an `expectedOutput` and exactly one of `fileId` (a single file) or `fileIds` (an ordered list of 2–50 files evaluated together as one [multifile extraction](/2026-02-09/extraction/multifile) run — JSON-schema extractors only). Items created with `fileIds` come back with `file: null` and their member files in `files`.

#### Python

```python
created = client.evaluation_set_items.create(
    eval_set.id,
    items=[
        {
            "file_id": uploaded.id,
            "expected_output": {"value": expected_value},
        },
    ],
)
print([item.id for item in created.evaluation_set_items])  # ["evi_..."]
```

#### TypeScript

```typescript
const created = await client.evaluationSetItems.create(evalSet.id, {
  items: [
    {
      fileId: uploaded.id,
      expectedOutput: { value: expectedValue },
    },
  ],
});
console.log(created.evaluationSetItems.map((item) => item.id)); // ["evi_..."]
```

#### Java

```java
import ai.extend.resources.evaluationsetitems.requests.EvaluationSetItemsCreateRequest;
import ai.extend.resources.evaluationsetitems.types.EvaluationSetItemsCreateRequestItemsItem;
import ai.extend.resources.evaluationsetitems.types.EvaluationSetItemsCreateResponse;
import ai.extend.types.ProvidedExtractOutput;
import ai.extend.types.ProvidedProcessorOutput;

EvaluationSetItemsCreateResponse created = client.evaluationSetItems().create(
    evalSet.getId(),
    EvaluationSetItemsCreateRequest.builder()
        .addItems(EvaluationSetItemsCreateRequestItemsItem.builder()
            .expectedOutput(ProvidedProcessorOutput.of(
                ProvidedExtractOutput.builder().value(expectedValue).build()))
            .fileId(uploaded.getId())
            .build())
        .build());
created.getEvaluationSetItems().forEach(item -> System.out.println(item.getId())); // "evi_..."
```

#### Go

```go
created, err := c.EvaluationSetItems.Create(ctx, evalSet.ID, &extend.EvaluationSetItemsCreateRequest{
	Items: []*extend.EvaluationSetItemsCreateRequestItemsItem{
		{
			FileID: extend.String(uploaded.ID),
			ExpectedOutput: &extend.ProvidedProcessorOutput{
				ProvidedExtractOutput: &extend.ProvidedExtractOutput{Value: &expectedValue},
			},
		},
	},
})
if err != nil {
	log.Fatal(err)
}
for _, item := range created.EvaluationSetItems {
	fmt.Println(item.ID) // "evi_..."
}
```

### Add a multifile item

If your extractor is designed for [multifile extraction](/2026-02-09/extraction/multifile), pass `fileIds` instead of `fileId`. The item is stored as an ordered package and, when the set runs, is evaluated as a single multifile extraction run with one `expectedOutput` covering all of the files.

* `fileIds` must contain 2–50 unique file IDs (use `fileId` for a single file), and exactly one of `fileId` or `fileIds` may be set per item.
* Multifile items are only supported on evaluation sets attached to a JSON-schema extractor. Classifier and splitter sets reject them.
* Multifile items are returned with `files` (an ordered array of file summaries) and `file: null`, matching the shape of a multifile `ExtractRun`. Single-file items have `file` set and `files: null`.

#### Python

```python
created = client.evaluation_set_items.create(
    eval_set.id,
    items=[
        {
            "file_ids": [contract.id, amendment.id],
            "expected_output": {"value": expected_value},
        },
    ],
)
item = created.evaluation_set_items[0]
print(item.file)  # None
print([f.id for f in item.files])  # ["file_...", "file_..."]
```

#### TypeScript

```typescript
const created = await client.evaluationSetItems.create(evalSet.id, {
  items: [
    {
      fileIds: [contract.id, amendment.id],
      expectedOutput: { value: expectedValue },
    },
  ],
});
const item = created.evaluationSetItems[0];
console.log(item.file); // null
console.log(item.files?.map((f) => f.id)); // ["file_...", "file_..."]
```

#### Java

```java
import java.util.List;

EvaluationSetItemsCreateResponse created = client.evaluationSetItems().create(
    evalSet.getId(),
    EvaluationSetItemsCreateRequest.builder()
        .addItems(EvaluationSetItemsCreateRequestItemsItem.builder()
            .expectedOutput(ProvidedProcessorOutput.of(
                ProvidedExtractOutput.builder().value(expectedValue).build()))
            .fileIds(List.of(contract.getId(), amendment.getId()))
            .build())
        .build());
ai.extend.types.EvaluationSetItem item = created.getEvaluationSetItems().get(0);
System.out.println(item.getFile().isPresent()); // false
item.getFiles().ifPresent(files -> files.forEach(f -> System.out.println(f.getId()))); // "file_..."
```

#### Go

```go
created, err := c.EvaluationSetItems.Create(ctx, evalSet.ID, &extend.EvaluationSetItemsCreateRequest{
	Items: []*extend.EvaluationSetItemsCreateRequestItemsItem{
		{
			FileIDs: []string{contract.ID, amendment.ID},
			ExpectedOutput: &extend.ProvidedProcessorOutput{
				ProvidedExtractOutput: &extend.ProvidedExtractOutput{Value: &expectedValue},
			},
		},
	},
})
if err != nil {
	log.Fatal(err)
}
item := created.EvaluationSetItems[0]
fmt.Println(item.File == nil) // true
for _, f := range item.Files {
	fmt.Println(f.ID) // "file_..."
}
```

#### cURL

```bash
curl -X POST https://api.extend.ai/evaluation_sets/ev_2LcgeY_mp2T5yPaEuq5Lw/items \
  -H "x-extend-api-version: 2026-02-09" \
  -H "Authorization: Bearer $EXTEND_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "items": [
      {
        "fileIds": ["file_xK9mLPqRtN3vS8wF5hB2cQ", "file_aB3cD4eF5gH6iJ7kL8mN9"],
        "expectedOutput": { "value": { "vendor_name": "Acme Corp", "total_amount": 1500.00 } }
      }
    ]
  }'
```

To bootstrap `expected_value` for a package, run the extractor with `package: { files: [{ id: ... }, ...] }` as shown in the [multifile quick start](/2026-02-09/extraction/multifile#quick-start) and correct the result, the same way you would for a single file.

### Maintaining items

* **List** what is in a set with `evaluationSetItems.list` (`evaluation_set_items.list` in Python).
* **Correct** an item's ground truth with `evaluationSetItems.update`, passing a new `expectedOutput`. The [Running Evaluation Sets](/2026-02-09/evaluation/running-evaluation-sets#update-an-items-expected-output) page shows this in all four languages. An item's files cannot be changed after creation; delete the item and recreate it instead.
* **Remove** an item with `evaluationSetItems.delete`.

> **Note**
>
> Items are scored against the extractor's schema at run time. If you change the schema (rename a field, add a required property), update the affected items' `expectedOutput` so the field paths still line up. Fields present in the schema but missing from `expectedOutput` count as incorrect.

## Via Extend Studio

There are two ways to build a set in the dashboard.

### Convert a batch run to an evaluation set

The most common way to create an evaluation set is by converting a batch run of reviewed outputs into a new evaluation set.

1. After running a batch of documents through your processor, review the results on the batch run page. Select the documents you'd like to add to an evaluation set and click "Add to evaluation set".

   You do not need to review all documents in the batch to create an evaluation set, but it is recommended. You can always update the expected outputs later.

![Batch run results page with several documents selected and the "Add to evaluation set" action highlighted](/_fern-files/_img/381ec9d78caa945529c4f27ce35086d0066823254bcd6e8e4c4d6a0db0153953.webp)

2. Either create a new evaluation set or add the selected files to an existing set.
   * For existing sets, select the evaluation set.
   * For new sets, enter a name and description.

![Dialog for choosing an existing evaluation set to add the selected files to](/_fern-files/_img/5d35bebd5f9fbce1934e384e8bb0a89c41e4af1d55b18222c372c7e6cd491bb0.webp)
![Dialog for naming and describing a new evaluation set](/_fern-files/_img/b1711423304502bd8fca7cb764662f69c6e22770dabe471d7cde343933dbc0a8.webp)

3. You are redirected to the evaluation set page where you can view and manage the documents and outputs in the set.

![Evaluation set detail page listing each item's file and expected output](/_fern-files/_img/7500f91d532dad0605a12a94fd08736bf2cc6d2b89578af5370257a740e2d572.webp)

### Create a new evaluation set from scratch

1. From the Evaluation tab in Extend, click the "Create Evaluation Set" button.

![Evaluation page in Extend Studio with the "Create Evaluation Set" button](/_fern-files/_img/be4d276b0953d6fca52a01cfe70475023fe7621dcdc8f66a7feeb044dff2b069.webp)

2. Enter a name and description and select a processor.

   You can run this evaluation set against any compatible processor, but you must select one to create the set. That processor becomes the set's default.

![Create evaluation set dialog with name, description, and processor fields](/_fern-files/_img/6141184677b0e7e23d84aaf1dbf2f4da317c0d02377f1cf3f200d7312eff8cd5.webp)

3. Add documents by clicking the "Add file" button. Upload a document, then validate the output once processing finishes (which may take several seconds).

![Add file dialog for uploading a document to an evaluation set](/_fern-files/_img/9811e7c65901bd2b275cad69169453466dee9a7f15e80d209cdd2db56ff4ad66.webp)

You can also add multiple documents at once from the runner results. See [Running Evaluation Sets](/2026-02-09/evaluation/running-evaluation-sets) for details.

## Reference

* [Create Evaluation Set](/2026-02-09/api-reference/endpoints/evaluation/create-evaluation-set), [Create Evaluation Set Items](/2026-02-09/api-reference/endpoints/evaluation/create-evaluation-set-item), [Update Evaluation Set Item](/2026-02-09/api-reference/endpoints/evaluation/update-evaluation-set-item), [Upload File](/2026-02-09/api-reference/endpoints/file/upload-file)
* [Running Evaluation Sets](/2026-02-09/evaluation/running-evaluation-sets): start a run, poll it, and read accuracy metrics
* [Processors](/2026-02-09/evaluation/processors): the saved, versioned entity an evaluation set is attached to
* [Extraction Response Format](/2026-02-09/extraction/response-format): how field paths in `value` are structured
* [Multifile Extraction](/2026-02-09/extraction/multifile): how packages of files are extracted together, and the `file` / `files` response shape that multifile items share