Create Evaluation Set Run | Extend Documentation

from extend_ai import Extend
client = Extend(
    token="YOUR_TOKEN_HERE",
    extend_api_version="2026-02-09",
)
client.evaluation_set_runs.create(
    evaluation_set_id="ev_2LcgeY_mp2T5yPaEuq5Lw",
    evaluation_set_item_ids=[
        "evi_kR9mNP12Qw4yTv8BdR3H"
    ],
)

{
  "object": "evaluation_set_run",
  "id": "bpr_Xj8mK2pL9nR4vT7qY5wZ",
  "evaluationSetId": "ev_2LcgeY_mp2T5yPaEuq5Lw",
  "entity": {
    "object": "extractor",
    "id": "ex_Xj8mK2pL9nR4vT7qY5wZ",
    "name": "Invoice Extractor",
    "createdAt": "2024-03-21T16:45:00Z",
    "updatedAt": "2024-03-21T16:45:00Z"
  },
  "entityVersion": {
    "object": "extractor_version",
    "id": "exv_xK9mLPqRtN3vS8wF5hB2cQ",
    "description": "Updated extraction fields for new invoice format",
    "version": "draft",
    "extractorId": "ex_Xj8mK2pL9nR4vT7qY5wZ",
    "createdAt": "2024-03-21T16:45:00Z",
    "createdBy": {
      "type": "user",
      "email": "matt@extend.app"
    }
  },
  "metrics": {
    "type": "EXTRACT",
    "accuracy": 0.8,
    "fieldMetrics": {
      "invoice_number": {
        "countTotal": 10,
        "countPresent": 10,
        "countExpected": 10,
        "countAccurate": 9,
        "accuracy": 0.9
      },
      "vendor.name": {
        "countTotal": 10,
        "countPresent": 8,
        "countExpected": 10,
        "countAccurate": 7,
        "accuracy": 0.7
      }
    },
    "meanLatencyMs": 100,
    "numFilesProcessed": 10,
    "numFilesTotal": 10,
    "numPages": 25,
    "p50LatencyMs": 90,
    "p90LatencyMs": 150,
    "p95LatencyMs": 200,
    "p99LatencyMs": 300,
    "numFiles": 10
  },
  "status": "PROCESSING",
  "options": {
    "fuzzyMatchFields": [
      "string"
    ],
    "excludeFields": [
      "string"
    ]
  },
  "createdAt": "2024-03-21T16:45:00Z",
  "updatedAt": "2024-03-21T16:45:00Z"
}

Create and start an async evaluation set run. The response returns the evaluation set run object with its initial status; use GET /evaluation_set_runs/{id} to poll for completion.

Evaluation set runs are currently supported for document processor evaluation sets.

Authentication

AuthorizationBearer

Bearer authentication of the form Bearer <token>, where token is your auth token.

Headers

Request

This endpoint expects an object.

evaluationSetIdstringRequired

The ID of the evaluation set to run.

entityobjectOptional

Optional processor and version to run against the evaluation set. If omitted, the evaluation set's processor is run at its draft version.

evaluationSetItemIdslist of stringsOptional

Optional list of evaluation set item IDs to run. If omitted, all items in the evaluation set are run.

Response

Evaluation set run created successfully

objectenum

The type of object. In this case, it will always be "evaluation_set_run".

Allowed values:

idstring

The unique identifier for this evaluation set run.

Example: "bpr_Xj8mK2pL9nR4vT7qY5wZ"

evaluationSetIdstring

The ID of the evaluation set that was run.

Example: "ev_2LcgeY_mp2T5yPaEuq5Lw"

entityobject

The extractor, classifier, or splitter that was run.

entityVersionobject

The version of the extractor, classifier, or splitter that was run.

metricsobject

Metrics for the evaluation set run. The shape depends on the entity type.

statusenum

The status of a batch run:

"PENDING" - The batch has been created and is waiting to be processed
"PROCESSING" - The batch is currently being processed
"PROCESSED" - All runs in the batch have completed successfully
"FAILED" - The batch failed to process
"CANCELLED" - The batch was cancelled

Allowed values:

optionsobject

The options for the evaluation set run.

createdAtstringformat: "date-time"

The time (in UTC) at which the object was created. Will follow the RFC 3339 format.

Example: "2024-03-21T16:45:00Z"

updatedAtstringformat: "date-time"

The time (in UTC) at which the object was last updated. Will follow the RFC 3339 format.

Example: "2024-03-21T16:45:00Z"

Errors

400

Bad Request Error

401

Unauthorized Error

402

Payment Required Error

403

Forbidden Error

404

Not Found Error

422

Unprocessable Entity Error

429

Too Many Requests Error

500

Internal Server Error