Confidence Scores

Extend uses machine learning models to extract information from documents. A confidence score tells you how ‘confident’ the model is about the accuracy of an extracted value. The score is presented as a number between 0 and 1, with values closer to 1 indicating higher confidence.

Below, you can see an example of an extraction with high confidence, and that all of the extracted values are correct.

Limitations and Best Practices

While confidence scores are a valuable tool in assessing the reliability of extracted data, it’s important to recognize their limitations to use them effectively.

Not a Guarantee of Accuracy

A high confidence score indicates a high probability of correctness, but it doesn’t guarantee accuracy. Even with a high score, there’s always a chance of errors, so critical data should always be cross-verified.

Context Matters

Confidence scores are calculated using models and their understanding of the data. They may not account for nuances or context that a human reviewer who understands your business would recognize. Because of this, it’s imperative to supply sufficient context on the value you want extracted so the AI can determine whether it is confident or not.

For example, if you name a field “company_name” on an extractor designed for invoices, the model may not be confident on which name you are asking for as there are a couple different company names on invoices.

Best Practices

• Provide clear field descriptions: Ambiguous field names can lead to lower confidence scores
• Test and iterate: Monitor confidence patterns in your specific use case and adjust thresholds accordingly

Common Usage Patterns

The following usage patterns apply:

Using Confidence Scores in Conditional Steps

Using conditional steps, you can handle a document differently depending on whether its extraction confidence score is above or below a certain ‘confidence threshold’. For instance, if you’re extracting important financial data, you might set a high threshold, accepting only data with confidence scores above 0.95. For less critical data, a lower threshold might be acceptable.

Review and Verification

Use confidence scores to prioritize data for review. Data points with low confidence scores might need manual verification or further investigation.

Data Filtering

In large datasets, you may want to filter out data below a certain confidence level to ensure the quality and reliability of your analysis.

Confidence Scores

We provide two types of confidence scores in the metadata object of processor run outputs:

• logprobsConfidence - Confidence from the language model based on token probabilities during generation
• ocrConfidence - Confidence from the OCR (Optical Character Recognition) system about text extraction accuracy

Both scores range from 0 to 1, with values closer to 1 indicating higher confidence.

Accessing Confidence Scores

Confidence scores are stored in the metadata object using path-like keys that correspond to your extracted data structure:

1
{
2
  "value": {
3
    "invoice_number": "INV-123",
4
    "line_items": [
5
      { "description": "Item A", "quantity": 2 }
6
    ]
7
  },
8
  "metadata": {
9
    "invoice_number": {
10
      "logprobsConfidence": 1.0,
11
      "ocrConfidence": 0.99
12
    },
13
    "line_items[0].description": {
14
      "logprobsConfidence": 0.98,
15
      "ocrConfidence": 0.95
16
    }
17
  }
18
}

Using Confidence in Conditional Steps

We provide several ways to access confidence scores in conditional steps:

Aggregate Confidence Access:

• {{extractionStepName.avgConfidence}} - Average confidence across all extracted fields
• {{extractionStepName.minConfidence}} - Minimum confidence across all extracted fields

Specific Field Confidence Access:

• {{extractionStepName.output.metadata.field_name.logprobsConfidence}} - Language model confidence for a specific field
• {{extractionStepName.output.metadata.field_name.ocrConfidence}} - OCR confidence for a specific field

Example Conditional Logic:

// Route to manual review if any field has low confidence
{{extractionStepName.minConfidence}} < 0.9
// Route to manual review if invoice number has low OCR confidence
{{extractionStepName.output.metadata.invoice_number.ocrConfidence}} < 0.95
// Route to automated processing if average confidence is high
{{extractionStepName.avgConfidence}} >= 0.98

Working with Arrays

Confidence scores are provided for arrays at multiple levels:

• Array-level confidence: Overall confidence for the entire array using the array field name
• Item-level confidence: Confidence for each object in the array using arrayName[index] notation
• Property-level confidence: Confidence for individual properties within array items using arrayName[index].propertyName notation

1
{
2
  "value": {
3
    "line_items": [
4
      { "description": "Item A", "quantity": 2 },
5
      { "description": "Item B", "quantity": 1 }
6
    ]
7
  },
8
  "metadata": {
9
    "line_items": {
10
      "logprobsConfidence": 0.98
11
    },
12
    "line_items[0]": {
13
      "logprobsConfidence": 0.99
14
    },
15
    "line_items[0].description": {
16
      "logprobsConfidence": 1.0,
17
      "ocrConfidence": 0.95
18
    },
19
    "line_items[0].quantity": {
20
      "logprobsConfidence": 1.0,
21
      "ocrConfidence": 0.98
22
    },
23
    "line_items[1].description": {
24
      "logprobsConfidence": 0.97,
25
      "ocrConfidence": 0.92
26
    }
27
  }
28
}

Programmatic Access

To access confidence scores programmatically:

1
const output = {
2
  value: { invoice_number: "INV-123" },
3
  metadata: {
4
    invoice_number: {
5
      logprobsConfidence: 1.0,
6
      ocrConfidence: 0.99
7
    }
8
  }
9
};
10
11
const invoiceNumberConfidence = output.metadata.invoice_number;
12
const logprobsScore = invoiceNumberConfidence?.logprobsConfidence;
13
const ocrScore = invoiceNumberConfidence?.ocrConfidence;
14
15
// For arrays
16
for (let i = 0; i < output.value.line_items.length; i++) {
17
  const itemPath = `line_items[${i}]`;
18
  const itemConfidence = output.metadata[itemPath];
19
  
20
  const descriptionPath = `${itemPath}.description`;
21
  const descriptionConfidence = output.metadata[descriptionPath];
22
  
23
  if (descriptionConfidence?.logprobsConfidence < 0.9) {
24
    console.log(`Low confidence for item ${i} description`);
25
  }
26
}