Confidence Scores

Extend uses machine learning models to extract information from documents. A confidence score tells you how ‘confident’ the model is about the accuracy of an extracted value. The score is presented as a number between 0 and 1, with values closer to 1 indicating higher confidence.

Below, you can see an example of an extraction with high confidence, and that all of the extracted values are correct.

Overview

The format and availability of confidence scores depends on your processor configuration type:

• JSON Schema Processors: Provide detailed confidence scores in a metadata object with logprobsConfidence and ocrConfidence values
• Fields Array Processors: Include confidence scores directly within each field’s result object (legacy configuration)

Common Usage Patterns

The following usage patterns apply to both JSON Schema and Fields Array processors:

Using Confidence Scores in Conditional Steps

Using conditional steps, you can handle a document differently depending on whether its extraction confidence score is above or below a certain ‘confidence threshold’. For instance, if you’re extracting important financial data, you might set a high threshold, accepting only data with confidence scores above 0.95. For less critical data, a lower threshold might be acceptable.

Review and Verification

Use confidence scores to prioritize data for review. Data points with low confidence scores might need manual verification or further investigation.

Data Filtering

In large datasets, you may want to filter out data below a certain confidence level to ensure the quality and reliability of your analysis.

JSON Schema Processors

JSON Schema processors provide two types of confidence scores in the metadata object of processor run outputs:

• logprobsConfidence - Confidence from the language model based on token probabilities during generation
• ocrConfidence - Confidence from the OCR (Optical Character Recognition) system about text extraction accuracy

Both scores range from 0 to 1, with values closer to 1 indicating higher confidence.

Accessing Confidence Scores

Confidence scores are stored in the metadata object using path-like keys that correspond to your extracted data structure:

1
{
2
  "value": {
3
    "invoice_number": "INV-123",
4
    "line_items": [
5
      { "description": "Item A", "quantity": 2 }
6
    ]
7
  },
8
  "metadata": {
9
    "invoice_number": {
10
      "logprobsConfidence": 1.0,
11
      "ocrConfidence": 0.99
12
    },
13
    "line_items[0].description": {
14
      "logprobsConfidence": 0.98,
15
      "ocrConfidence": 0.95
16
    }
17
  }
18
}

Using Confidence in Conditional Steps

JSON Schema processors provide several ways to access confidence scores in conditional steps:

Aggregate Confidence Access:

• {{extractionStepName.avgConfidence}} - Average confidence across all extracted fields
• {{extractionStepName.minConfidence}} - Minimum confidence across all extracted fields

Specific Field Confidence Access:

• {{extractionStepName.output.metadata.field_name.logprobsConfidence}} - Language model confidence for a specific field
• {{extractionStepName.output.metadata.field_name.ocrConfidence}} - OCR confidence for a specific field

Example Conditional Logic:

// Route to manual review if any field has low confidence
{{extractionStepName.minConfidence}} < 0.9
// Route to manual review if invoice number has low OCR confidence
{{extractionStepName.output.metadata.invoice_number.ocrConfidence}} < 0.95
// Route to automated processing if average confidence is high
{{extractionStepName.avgConfidence}} >= 0.98

Working with Arrays

For JSON Schema processors, confidence scores are provided for arrays at multiple levels:

• Array-level confidence: Overall confidence for the entire array using the array field name
• Item-level confidence: Confidence for each object in the array using arrayName[index] notation
• Property-level confidence: Confidence for individual properties within array items using arrayName[index].propertyName notation

1
{
2
  "value": {
3
    "line_items": [
4
      { "description": "Item A", "quantity": 2 },
5
      { "description": "Item B", "quantity": 1 }
6
    ]
7
  },
8
  "metadata": {
9
    "line_items": {
10
      "logprobsConfidence": 0.98
11
    },
12
    "line_items[0]": {
13
      "logprobsConfidence": 0.99
14
    },
15
    "line_items[0].description": {
16
      "logprobsConfidence": 1.0,
17
      "ocrConfidence": 0.95
18
    },
19
    "line_items[0].quantity": {
20
      "logprobsConfidence": 1.0,
21
      "ocrConfidence": 0.98
22
    },
23
    "line_items[1].description": {
24
      "logprobsConfidence": 0.97,
25
      "ocrConfidence": 0.92
26
    }
27
  }
28
}

Programmatic Access

To access confidence scores programmatically in JSON Schema processors:

1
const output = {
2
  value: { invoice_number: "INV-123" },
3
  metadata: {
4
    invoice_number: {
5
      logprobsConfidence: 1.0,
6
      ocrConfidence: 0.99
7
    }
8
  }
9
};
10
11
const invoiceNumberConfidence = output.metadata.invoice_number;
12
const logprobsScore = invoiceNumberConfidence?.logprobsConfidence;
13
const ocrScore = invoiceNumberConfidence?.ocrConfidence;
14
15
// For arrays
16
for (let i = 0; i < output.value.line_items.length; i++) {
17
  const itemPath = `line_items[${i}]`;
18
  const itemConfidence = output.metadata[itemPath];
19
  
20
  const descriptionPath = `${itemPath}.description`;
21
  const descriptionConfidence = output.metadata[descriptionPath];
22
  
23
  if (descriptionConfidence?.logprobsConfidence < 0.9) {
24
    console.log(`Low confidence for item ${i} description`);
25
  }
26
}

Fields Array Processors

Fields Array processors include confidence scores directly within each field’s result object. The specific format and availability may vary based on your processor configuration.

Using Confidence in Conditional Steps

Fields Array processors can use confidence scores in conditional steps for routing decisions:

Working with Arrays

For Fields Array processors, confidence scores are included within the array field result structure. To view these confidences in the UI, hover over the cell the value is contained in.

Low confidence values are highlighted yellow. When reviewing an array with highlighted low confidence values:

Low confidence values in arrays might indicate that the model doesn’t have enough context to extract the value correctly or that the model is confused by the format of the document. In these cases, it may be helpful to add more context to your extraction field descriptions. If this does not remedy the issue, reach out to the Extend team.

Programmatic Access

Fields Array processors include confidence scores directly within each field’s result object. The specific format and availability may vary based on your processor configuration.

Limitations and Best Practices

While confidence scores are a valuable tool in assessing the reliability of extracted data, it’s important to recognize their limitations to use them effectively. These considerations apply to both JSON Schema and Fields Array processors.

Not a Guarantee of Accuracy

A high confidence score indicates a high probability of correctness, but it doesn’t guarantee accuracy. Even with a high score, there’s always a chance of errors, so critical data should always be cross-verified.

Context Matters

Confidence scores are calculated using models and their understanding of the data. They may not account for nuances or context that a human reviewer who understands your business would recognize. Because of this, it’s imperative to supply sufficient context on the value you want extracted so the AI can determine whether it is confident or not.

For example, if you name a field “company_name” on an extractor designed for invoices, the model may not be confident on which name you are asking for as there are a couple different company names on invoices.

Best Practices

• Provide clear field descriptions: Ambiguous field names can lead to lower confidence scores
• Test and iterate: Monitor confidence patterns in your specific use case and adjust thresholds accordingly