Evaluation

Evaluation Overview

Evaluation sets let you test the accuracy of your document processors reliably and repeatedly. An evaluation set is attached to a processor and holds items: files paired with the output you expect. An evaluation set run scores one processor version against those items and returns accuracy metrics, so you can verify that a configuration change improved results before you publish it. Evaluation sets can also be run against a workflow to score its end-to-end output; the pages below focus on processors.

Evaluation is fully available through the API and in Extend Studio:

TaskAPIStudio
Create a set and add items with expected outputsevaluationSets.create, evaluationSetItems.createConvert a reviewed batch run, or add files one at a time
Run a set against a processor versionevaluationSetRuns.create, then poll evaluationSetRuns.retrieveClick “Run” from the processor runner or the set’s page
Read resultsAggregate metrics on the run (overall and per-field / per-class accuracy)Metrics summary, per-document diffs, CSV export, run comparison
Correct ground truthevaluationSetItems.update”Update” on a document’s result
Creating Evaluation Sets

Create a set, upload files, and add items with expected outputs, in Python, TypeScript, Java, Go, or Studio.

Running Evaluation Sets

Start a run, poll it to completion, and read accuracy metrics.

The evaluation loop

  1. Create an evaluation set containing examples that represent the range of documents your processor needs to handle. Aim for coverage of layouts, vendors, and edge cases rather than volume.
  2. Run it against the processor version you are working on (the draft, or a published version).
  3. Review the results to confirm outputs match expectations and to find common errors. Per-field accuracy tells you which parts of a schema need work.
  4. Iterate on the processor configuration (field descriptions, rules, base processor) and rerun. Compare runs to confirm the change helped. Composer automates this step using your evaluation sets.
  5. Publish the improved version once the set passes your bar, and pin it in your workflows.

In Extend Studio

The Evaluation page in the Studio section lists your evaluation sets with options to create new sets and start runs.

Evaluation page in Extend Studio listing evaluation sets with their processors and latest run accuracy

Reference