Evaluation

Creating Evaluation Sets

An evaluation set is a collection of items: each item pairs a file with the output you expect a processor to produce for it. You create the set once against a primary extractor, classifier, or splitter, add items over time, and then run the set to measure accuracy.

You can build a set entirely through the API, entirely in Extend Studio, or mix the two (for example, create items via the API and review them in Studio). Evaluation sets can also be run against workflows; this page covers sets for processors.

Via the API

The full flow is four calls: create the set, upload a file, produce the expected output, and add the item. Use this when you want an evaluation loop in code, for example to rebuild a set from a folder of reviewed documents.

1

Create the evaluation set

entityId is the primary processor the set is for. You can later run the set against any compatible processor, but every set has one default.

from extend_ai import Extend
client = Extend()
eval_set = client.evaluation_sets.create(
    name="Invoice extraction — Q3 vendors",
    description="Reviewed vendor invoices used to gate extractor releases",
    entity_id="ex_Xj8mK2pL9nR4vT7qY5wZ",
)
print(eval_set.id)  # "ev_..."
2

Upload the file

Items reference files by Extend file ID (file_...). Upload each document once; you can reuse the same file ID in multiple evaluation sets.

with open("invoices/acme-2026-07.pdf", "rb") as f:
    uploaded = client.files.upload(file=f)
print(uploaded.id)  # "file_..."
3

Produce the expected output

The expected output is the ground truth the run is scored against. If you already have reviewed ground truth for the document (from an existing system, a spreadsheet, or a human review), map it to the shape below and use it directly; this is the preferred source because it is independent of the processor you are evaluating. The shape depends on the processor type:

ProcessorexpectedOutput shape
Extractor{ "value": { ...fields conforming to the extractor's schema... } } — nest the object under value and include every property in the schema
Classifier{ "id": "<classification id>", "type": "<classification name>" }
Splitter{ "splits": [{ "classificationId", "type", "startPage", "endPage" }, ...] }

If you do not have ground truth yet, bootstrap it from a run: process the file, correct any mistakes, and save the corrected result. For an extractor, the run’s output already matches the schema’s field names and types:

run = client.extract(
    file={"id": uploaded.id},
    extractor={"id": "ex_Xj8mK2pL9nR4vT7qY5wZ"},
)
# Correct anything the extractor got wrong before saving it as ground truth
expected_value = dict(run.output.value)
expected_value["total"] = {"amount": 1249.00, "iso_4217_currency_code": "USD"}
4

Add the item

You can add up to 100 items in one call. Each item needs a fileId and an expectedOutput.

created = client.evaluation_set_items.create(
    eval_set.id,
    items=[
        {
            "file_id": uploaded.id,
            "expected_output": {"value": expected_value},
        },
    ],
)
print([item.id for item in created.evaluation_set_items])  # ["evi_..."]

Maintaining items

  • List what is in a set with evaluationSetItems.list (evaluation_set_items.list in Python).
  • Correct an item’s ground truth with evaluationSetItems.update, passing a new expectedOutput. The Running Evaluation Sets page shows this in all four languages.
  • Remove an item with evaluationSetItems.delete.

Items are scored against the extractor’s schema at run time. If you change the schema (rename a field, add a required property), update the affected items’ expectedOutput so the field paths still line up. Fields present in the schema but missing from expectedOutput count as incorrect.

Via Extend Studio

There are two ways to build a set in the dashboard.

Convert a batch run to an evaluation set

The most common way to create an evaluation set is by converting a batch run of reviewed outputs into a new evaluation set.

  1. After running a batch of documents through your processor, review the results on the batch run page. Select the documents you’d like to add to an evaluation set and click “Add to evaluation set”.

    You do not need to review all documents in the batch to create an evaluation set, but it is recommended. You can always update the expected outputs later.

Batch run results page with several documents selected and the "Add to evaluation set" action highlighted

  1. Either create a new evaluation set or add the selected files to an existing set.
    • For existing sets, select the evaluation set.
    • For new sets, enter a name and description.

Dialog for choosing an existing evaluation set to add the selected files to Dialog for naming and describing a new evaluation set

  1. You are redirected to the evaluation set page where you can view and manage the documents and outputs in the set.

Evaluation set detail page listing each item's file and expected output

Create a new evaluation set from scratch

  1. From the Evaluation tab in Extend, click the “Create Evaluation Set” button.

Evaluation page in Extend Studio with the "Create Evaluation Set" button

  1. Enter a name and description and select a processor.

    You can run this evaluation set against any compatible processor, but you must select one to create the set. That processor becomes the set’s default.

Create evaluation set dialog with name, description, and processor fields

  1. Add documents by clicking the “Add file” button. Upload a document, then validate the output once processing finishes (which may take several seconds).

Add file dialog for uploading a document to an evaluation set

You can also add multiple documents at once from the runner results. See Running Evaluation Sets for details.

Reference