Scenario: Structured Data Extraction

Scenario Description

You're building a structured data extraction system that pulls information from unstructured documents (invoices, contracts, research papers, emails) and outputs validated, schema-compliant JSON. The challenge is maintaining high accuracy across diverse document formats while handling edge cases gracefully. The extracted data flows downstream into databases, APIs, and dashboards, so invalid or incomplete extraction breaks everything downstream.

This scenario requires mastering JSON schema design, tool_use configuration with tool_choice settings, validation-retry loops, and batch processing for cost efficiency. You'll design schemas with required fields, nullable fields, enums with "other" + detail fields, and special "unclear" values for genuinely ambiguous cases. You'll implement few-shot examples that show the model how to handle varied document structures (e.g., invoices with different line-item formats).

Cost and reliability matter equally. The Batch API gives 50% cost savings but introduces 24-hour latency, making it ideal for overnight extraction reports. Synchronous extraction is faster but costs more. You'll also implement human review workflows with field-level confidence scores, allowing reviewers to focus on borderline cases rather than obvious extractions.

Domains Tested

Key Concepts to Study

Prompt Engineering & Structured Output

tool_use with JSON schemas: leveraging the Tools API to guarantee structured, schema-compliant output
tool_choice configuration: "auto" (model decides when to extract), "any" (model must use a tool), forced tool name (e.g., always use extract_invoice_data)
Schema design patterns: required vs. optional/nullable fields, enums with "other" + detail fields for extensibility, "unclear" values for ambiguous data
Few-shot examples: demonstrating extraction patterns for varied document structures (e.g., invoices with 1-line items vs. 50-line items, different date formats)
Error feedback in validation-retry loops: when extraction fails schema validation, feeding specific error messages back to the model for correction

Context Management & Reliability

Batch processing with Message Batches API: grouping extraction requests for 24-hour processing, custom_id for tracking, 50% cost savings
Human review workflows: assigning confidence scores to extracted fields so reviewers prioritize high-uncertainty extractions
Stratified random sampling for accuracy measurement: testing extraction quality on representative samples rather than all documents
Error handling and fallback strategies: what to do when a field can't be extracted (leave null, set a default, escalate for human review)

Study Tips for This Scenario

Design Schemas with Edge Cases in Mind: Before writing extraction prompts, enumerate edge cases: missing fields, ambiguous date formats, field aliases (Invoice # vs. Invoice Number), multi-line text fields. For each edge case, decide: required (fail if missing), nullable (accept null), or fallback (use a default). Encode these decisions into your schema as required arrays, type: ["string", "null"], and comment fields explaining expectations.
Write Few-Shot Examples for Varied Document Styles: Provide 2–3 extraction examples showing different document layouts (minimal invoice vs. detailed invoice, single-page vs. multi-page contract). For each example, show both the input (document excerpt) and expected output (JSON). This anchors the model's behavior and reduces hallucination for unfamiliar formats.
Implement Validation-Retry Logic with Specific Error Feedback: After extracting, validate output against the schema. If validation fails (e.g., amount is string instead of number), don't just re-run extraction blindly. Instead, send a message like: "The extracted amount '€1.234,56' is a string but must be a number. Re-extract and convert to numeric format, using '.' as decimal separator." Specific error feedback reduces retry loops.
Use Batch API for High-Volume, Offline Extraction: For batch jobs (end-of-month invoice processing, document archival), use the Batches API. Submit 100+ extraction requests with custom_ids, collect results after 24 hours, and pay 50% less. Implement a callback system that, when results arrive, triggers validation and human review workflows. Reserve synchronous extraction for real-time, low-volume use cases.