Running well maintained and up to date evaluation sets is one of the best ways to ensure that your AI document processors are performing as expected and to identify areas for improvement, while iterating.
Follow this guide to learn more about how to run and maintain evaluation sets in Extend.
The default options are the processor and version you selected in the run ui. You can however update the version, or the actual processor before running here. See below for advanced option configuration
Click “Run Evaluation” to start the evaluation. You will be redirected to the evaluation run page where you can view the progress of the evaluation.
You can see a metrics summary at the top of the page, and a list of documents that were run and their results below.
Each processor type will have different metrics views. For instance a classification processor will show a per type accuracy distribution and a confusion matrix-like view.
There are a few advanced options you can configure when running an eval set that can affect how metrics are calculated.
If you want to exclude a field from the evaluation run’s metrics, simply uncheck the checkbox next to the field. The per document accuracy and overall accuracy will not take this field into account.
For string type fields, you can configure a custom matcher. There are four available to toggle from: strict, fuzzy, LLM judge, and vector.
The default matcher - checks for strict string equality and is case sensitive.
Uses fuzzy matching distance which can allow for small errors between the actual and expected output. You can configure a threshold for what is considered a match (values above the threshold are a match). Uses a modified Levenshtein distance algorithm.
Recommended if you expect some variability, but only by a few characters.
Passes the actual and expected output to an LLM and uses it to determine if they are semantically the same. By default, it can handle things like date formats (1/1/2025 vs Jan 1st 2025), abbreviations (St. vs Street), and numericals (1,000,000 vs one million). We highly recommend adding a custom instruction that acts as a rubric for the model to determine matches.
For example: Return true if and only if the expected and actual output have the same address, but you can ignore added or removed text.
Recommended if the extracted values are long sentences / paragraphs, or if you expect variability and want to define custom rules for matching
Embeds both the actual and expected value, and calculates the cosine similarity. Values that are more semantically similar will have a higher score (“house” and “home”). Values that are not semantically similar will have a lower score (“hot” and “cold”). You can configure a threshold for what is considered a match (values above the threshold are a match).
Recommended if you are trying to match based on the value’s meaning
For booleans, null and false will be treated as a match. For numbers, 0 and null will be treated as a match. For currency $0.00 and null will be a match.
The rate limit simply caps how many runs will be executed in a time period by this eval set run.
Clearing the pre-processing cache will re-run some pre-processing operations like chunking, metadata extraction, etc.
To get a general idea of how your eval set is performing, you can compare accuracy across runs. Select the runs to compare and click “Compare runs” in the runs tab of the eval set home page.
In this dialog, you can see color coded accuracy metrics, and whether they increased or decreased between runs. Hover over a column to see the exact change in percentage.
