Migrating to JSON Schema

Current Config Structure

This section gives some background on the current config structure and the new JSON Schema config structure. If you’d like to jump to migrating to the new JSON Schema config structure, you can go straight to the Migrating to JSON Schema section.

If your organization started using Extend before April 2025, you likely have been using the Fields Array config type. You can tell whether a processor is using the Fields Array config type by looking at processor details in the Studio. If you don’t see this section, your processor is using the JSON Schema config type and you do not have access to the legacy “Fields Array” config type.

If you’re processor is using the Fields Array config type, the config object in processor has a fields array that contains the fields for the processor. Here is an example config object of this type:

1{
2  "type": "EXTRACT",
3  "fields": [
4    {
5      "id": "invoice_number",
6      "name": "invoice_number",
7      "type": "string",
8      "description": "The unique identifier for this invoice"
9    },
10    {
11      "id": "amount",
12      "name": "amount",
13      "type": "currency",
14      "description": "The total amount of the invoice"
15    }
16  ]
17  // other fields...
18}

This schema has worked well, however since releasing it, the industry has standardized around JSON Schema as the way response schemas are defined. To make our processors easier to use for developers, we are moving to JSON Schema as the way schemas are defined for processors.

New JSON Schema Config Structure

A JSON Schema config object equivalent of the above example is:

1{
2  "type": "EXTRACT",
3  "schema": {
4    "type": "object",
5    "properties": {
6      "invoice_number": {
7        "type": ["string", "null"],
8        "description": "The unique identifier for this invoice"
9      },
10      "amount": {
11        "type": "object",
12        "properties": {
13          "value": {
14            "type": ["number", "null"]
15          },
16          "iso_4217_currency_code": {
17            "type": ["string", "null"]
18          }
19        },
20        "required": ["value", "iso_4217_currency_code"],
21        "additionalProperties": false
22      }
23    },
24    "required": ["invoice_number", "amount"],
25    "additionalProperties": false
26  }
27  // other fields...
28}

You’ll notice that instead of the fields array, we have a schema object. This object is a JSON Schema object that describes the shape of the output you will receive from the processor.

For more information on the JSON Schema config structure, please see the JSON Schema Config section of the API Reference.

Current Output Structure

The current output structure for Extraction processors is an object with the field names as keys and the values inside an object with the following properties:

  • id: The unique identifier for the field
  • type: The type of the field
  • value: The value of the field
  • confidence: The confidence score of the field
  • insights: The insights for the field
  • references: The references for the field

Here is an example of the output:

1{
2  "invoice_number": {
3    "id": "invoice_number",
4    "type": "string",
5    "value": "36995",
6    "confidence": 0.98,
7    "insights": [
8      {
9        "type": "reasoning",
10        "content": "The invoice number is clearly labeled as 'Invoice #36995' at the top right of the document, making it straightforward to extract."
11      }
12    ],
13    "references": [
14      {
15        "page": 1,
16        "boundingBoxes": [
17          {
18            "left": 296.73359999999997,
19            "top": 40.888799999999996,
20            "right": 386.4168,
21            "bottom": 52.1712
22          }
23        ],
24        "referenceText": "Invoice #36995"
25      }
26    ]
27  },
28  "amount": {
29    "id": "amount",
30    "type": "number",
31    "value": 15735.1,
32    "confidence": 0.98,
33    "insights": [
34      {
35        "type": "reasoning",
36        "content": "The total amount is shown as '$15,735.1' in both the table summary and the bottom right of the document. The currency symbol '$' and the US address indicate the currency is USD. The value is numeric and matches the required format."
37      }
38    ],
39    "references": [
40      {
41        "page": 1,
42        "boundingBoxes": [
43          {
44            "left": 430.164,
45            "top": 722.772,
46            "right": 467.27279999999996,
47            "bottom": 722.8296
48          }
49        ],
50        "referenceText": "TOTAL  $15,735.1"
51      }
52    ]
53  }
54}

In this output, the metadata like confidence, insights, and references are nested inside each field’s object right next to the value. The benefit of this is it’s very easy to access the metadata for a specific field. The downside is that it doesn’t work very well for recursive fields like arrays and objects.

New JSON Schema Output Structure

The output structure for JSON Schema processors is composed of two properties: value and metadata.

The value object is the actual data extracted from the document which conforms to the JSON Schema defined in the processor config.

The metadata object holds details like confidence scores and citations for the extracted data. It uses keys that represent the path to the corresponding data within the value object. For instance, if your data has value.line_items[0].name, the metadata specifically for that name field will be found using the key ‘line_items[0].name’ within the metadata object. For more information on the metadata object, please see the Accessing Metadata section of the API Reference.

Below is an example of the output you will receive from a JSON Schema processor:

1{
2  "value": {
3    "amount": {
4      "amount": 15735.1,
5      "iso_4217_currency_code": "USD"
6    },
7    "invoice_number": "36995"
8  },
9  "metadata": {
10    "amount": {
11      "insights": [
12        {
13          "type": "reasoning",
14          "content": "The total amount is shown as '$15,735.1' in both the table summary and the bottom right of the document. The currency symbol '$' and the US address indicate the currency is USD. The value is numeric and matches the required format."
15        }
16      ],
17      "citations": [
18        {
19          "page": 1,
20          "polygon": [
21            {
22              "x": 430.164,
23              "y": 722.772
24            },
25            {
26              "x": 467.27279999999996,
27              "y": 722.8296
28            },
29            {
30              "x": 467.2584,
31              "y": 731.6351999999999
32            },
33            {
34              "x": 430.1496,
35              "y": 731.5776
36            }
37          ],
38          "referenceText": "TOTAL  $15,735.1"
39        }
40      ],
41      "ocrConfidence": 0.992,
42      "logprobsConfidence": 1
43    },
44    "invoice_number": {
45      "insights": [
46        {
47          "type": "reasoning",
48          "content": "The invoice number is clearly labeled as 'Invoice #36995' at the top right of the document, making it straightforward to extract."
49        }
50      ],
51      "citations": [
52        {
53          "page": 1,
54          "polygon": [
55            {
56              "x": 296.73359999999997,
57              "y": 40.888799999999996
58            },
59            {
60              "x": 386.4168,
61              "y": 40.464000000000006
62            },
63            {
64              "x": 386.4744,
65              "y": 52.1712
66            },
67            {
68              "x": 296.7912,
69              "y": 52.596000000000004
70            }
71          ],
72          "referenceText": "Invoice #36995"
73        }
74      ],
75      "ocrConfidence": 0.986,
76      "logprobsConfidence": 1
77    }
78  }
79}

The benefit of this output structure is that it’s very easy to access the data for a specific field and it should be easy to ingest it as a typed object because it conforms to the JSON Schema defined in the processor config.

The Typescript types for the output are the following:

TypeScript Types
1type ExtractionOutput = {
2  value: ExtractionValue;
3  metadata: ExtractionMetadata;
4};
5
6type ExtractionValue = Record<string, any>; // Conforms to the schema defined in the processor config
7type ExtractionMetadata = {
8  [key: string]: ExtractionMetadataEntry | undefined;
9};
10
11type ExtractionMetadataEntry {
12  ocrConfidence?: number | null;
13  logprobsConfidence: number | null;
14  citations?: Citation[];
15  insights?: OutputInsight[];
16}
17
18type Citation = {
19  page?: number;
20  referenceText?: string | null;
21  polygon?: Point[];
22};
23
24type Point {
25  /**
26   * x coordinate - relative from the left side of the page
27   */
28  x: number;
29  /**
30   * y coordinate - relative from the top of the page
31   */
32  y: number;
33}
34
35type Insight = {
36  type: "reasoning";
37  content: string;
38};

Migrating to JSON Schema

To migrate a processor from the legacy Fields Array config type to the JSON Schema config type, you will need to:

  1. Go to the processor in Studio that you’d like to migrate.
  2. Click the button with the three vertical dots in the top right corner to open the settings menu.

  1. Click “Migrate to JSON Schema”. This will open a modal where you can select the version and choose the name for the new processor. Click “Migrate to JSON Schema”. This will create a new processor with the fields array replaced with a JSON Schema config object.

Please share any feedback you have on the new JSON Schema config type and output structure with us on Slack!