04 - Structured Output & Tool Calling

Scope

How to make an LLM emit machine-parseable JSON on demand, and how to let it act — call your functions, hit APIs, run an agentic loop. Covers the spectrum from “ask nicely for JSON” → JSON mode → Structured Outputs (json_schema + strict:true) → constrained decoding that masks tokens during generation. Then the tool-use side: tool specs, tool_choice, parallel calls, the agentic loop, and the library landscape (Pydantic, Instructor, Outlines, BAML). For how the model picks tokens under the hood, see 02 - Decoding, Sampling & Reasoning Controls; this note is about constraining that choice to a contract.

1. The concept: turning free text into a contract

A raw LLM emits a token stream. Most software wants a typed value — a record to store, an argument to pass, a field to render. The gap between “plausible prose” and “valid {"name": str, "age": int}” is the whole problem here, and there’s a ladder of increasingly strong guarantees:

Rung	Mechanism	Guarantee	Failure mode
Prompt-and-pray	”Reply with JSON only.”	none	preamble, markdown fences, trailing prose, invalid JSON
JSON mode	provider flag forces syntactically valid JSON	parses without error	right syntax, wrong shape — keys/types unconstrained
Structured Outputs	`json_schema` + `strict:true`	matches your schema’s shape/types	shape correct, values not validated for meaning
Constrained decoding	per-token logit masking against a grammar/FSM	output is grammar-valid by construction	grammar can’t express semantic rules; may distort reasoning

The load-bearing distinction

Schema guarantees SHAPE, not CORRECTNESS. strict:true will hand you {"email": "not-an-email", "quantity": -5} — every type is right, every required field present, and the values are garbage. Structure validation is necessary, never sufficient. The correctness check lives in your code, after the model returns.

2. JSON mode vs Structured Outputs

These get conflated constantly; they are different features.

JSON mode (the older feature) only promises the output is valid JSON. It does not know your schema. You still describe the desired shape in the prompt and hope. You can get {"answer": 42} when you wanted {"result": {"value": 42}}. Without an explicit instruction to produce JSON in the prompt, JSON mode can also loop emitting whitespace until it hits the token cap.

Structured Outputs (the 2024+ feature, now the production default) takes an actual JSON Schema and constrains generation to it. With OpenAI’s response_format: {type: "json_schema", json_schema: {..., strict: true}}, the model cannot return a non-conforming object — every required field is present, every type matches, every enum value is legal.

# OpenAI Structured Outputs — strict schema adherence
resp = client.responses.parse(
    model="...",
    input=[{"role": "user", "content": "Extract: Jane (jane@co.com), Enterprise plan"}],
    text_format=ContactInfo,   # a Pydantic model → JSON Schema under the hood
)
contact = resp.output_parsed     # already a validated ContactInfo instance

Reach for Structured Outputs, not JSON mode

If your provider supports json_schema + strict, use it. JSON mode is the weaker legacy tier — it solves “is it JSON” but leaves “is it my JSON” to chance. The only reason to use plain JSON mode today is a model/endpoint that doesn’t support strict schemas.

Anthropic’s equivalent. Claude exposes structured outputs via output_config: {format: {type: "json_schema", schema: {...}}} on the Messages API (the older top-level output_format is deprecated — use output_config.format). The SDK’s client.messages.parse() validates the response against a Pydantic/Zod model for you. It also offers strict tool use (strict: true on a tool definition) to guarantee tool arguments validate exactly.

# Anthropic — strict schema on the response
resp = client.messages.create(
    model="claude-opus-4-8",
    max_tokens=1024,
    messages=[{"role": "user", "content": "Extract: John (john@example.com), Enterprise plan"}],
    output_config={
        "format": {
            "type": "json_schema",
            "schema": {
                "type": "object",
                "properties": {
                    "name":  {"type": "string"},
                    "email": {"type": "string"},
                    "plan":  {"type": "string"},
                },
                "required": ["name", "email", "plan"],
                "additionalProperties": False,
            },
        }
    },
)

3. JSON Schema 2020-12 as the contract — and the provider subset

The contract language is JSON Schema (the 2020-12 draft is current). It expresses types, enum, const, required, anyOf/allOf, $ref/$defs, string formats (date-time, email, uuid, …), and value constraints (minimum, maxLength, pattern).

Pros

Standard, language-agnostic, tool-rich (every language has a validator).
Pydantic (Python) and Zod (TS) generate it from native types — you write the model once, get the schema for free.

Cons / the catch: providers implement only a subset for strict mode. The constraint engine that guarantees adherence can’t enforce every keyword cheaply, so the strict subset is narrower than full JSON Schema.

OpenAI strict-mode subset rules (the ones that bite)

For strict: true, OpenAI requires:

The root must be an object (not anyOf/oneOf at the top level).

additionalProperties: false on every object in the schema.

Every property must be in required. There are no optional fields — model an “absent” field as a union with null ({"type": ["string", "null"]} / Pydantic Optional), not by omitting it from required.

Value constraints are silently dropped or rejected: minimum/maximum/multipleOf, minLength/maxLength/pattern, most format semantics, complex array bounds. The schema enforces types and presence, not valid ranges or patterns.

Anthropic’s strict subset is similar in spirit: additionalProperties: false + required, basic types + enum/const/anyOf/$ref, the common string formats, but no numeric constraints (minimum/maximum), no string-length/pattern constraints, and no recursive schemas. The Python/TS SDKs quietly strip unsupported keywords before sending and re-validate them client-side.

Risk mitigation: a post-validation layer is mandatory

Because the strict subset can’t express pattern, minimum, or format enforcement, put a real validator after the model call. The schema gives you the right shape; your Pydantic/Zod model (with the full constraints) gives you the right values. Pattern:

Strict schema → model returns shape-correct JSON.

Full Pydantic model (with EmailStr, conint(gt=0), regex) validates the values.

On failure → reject, repair, or re-ask.

Never trust a strict-mode field’s value just because the request succeeded.

4. Constrained / grammar-guided decoding (the strongest rung)

This is the mechanism behind the best Structured Outputs implementations, and what open-weight stacks use directly.

Concept. At every generation step the model produces logits over the whole vocabulary. A grammar engine (driven by an FSM for regex/simple schemas, or a pushdown automaton for nested/recursive JSON) computes which tokens could legally continue the partial output. Every illegal token’s logit is set to -∞ (masked) before sampling. The model literally cannot emit a token that breaks the grammar — validity is guaranteed during generation, so there is no parse failure, no retry, no fallback parser.

logits ──▶ [grammar engine: which tokens keep the output valid?] ──▶ mask illegal → -∞ ──▶ sample
              ▲                                                                              │
              └────────────────── advance FSM/PDA state with chosen token ◀─────────────────┘

This is the same logit-masking machinery as logit_bias, but driven automatically by a schema/grammar instead of hand-set per-token biases — see 02 - Decoding, Sampling & Reasoning Controls.

Two philosophies, one outcome

Approach	When validity is enforced	Cost model	Examples
Constrained decoding (“during generation”)	every token, at inference	needs logit access → server-side or open-weight; near-zero per-token overhead once compiled	Outlines, XGrammar, llguidance, vLLM/SGLang/TensorRT-LLM backends
Validate-and-retry (“after generation”)	after a full response, in your code	extra LLM calls on failure (latency + tokens)	Instructor

Constrained decoding (during generation). XGrammar is, as of 2026, the default structured-generation backend for vLLM, SGLang, and TensorRT-LLM — sub-40µs per token with near-zero overhead on JSON, thanks to grammar caching. Outlines pioneered the FSM approach but compiles complex schemas slowly (seconds to minutes for big schemas), which hurts on first-use of an uncached schema. These run where you have logit access: self-hosted open-weight models, or as the engine inside a provider’s Structured Outputs.

Validate-and-retry (Instructor). Instructor sits on top of a normal chat API. It injects your Pydantic schema, parses the response, runs full Pydantic validation (constraints included), and on failure re-asks the model with the validation error as feedback. It works against any provider (no logit access needed) and enforces the full schema — including the pattern/minimum/format rules the strict subset drops — at the price of extra round-trips when the model misses.

Which to pick

Hosted API + want guaranteed shape, zero retries → provider Structured Outputs (json_schema + strict). The provider runs constrained decoding for you.

Self-hosting open weights → Outlines or XGrammar directly (XGrammar for complex/repeated schemas; both via vLLM).

Need full value constraints across any provider, tolerate occasional retries → Instructor.

Constrained decoding can't enforce meaning either

Masking guarantees grammar-validity, not semantic correctness. A grammar that accepts any string for email will happily let the model emit "asdf". The grammar pins structure; values still need a semantic check. Same lesson as §3 — only the layer where it’s enforced differs.

5. Tool / function calling: the spec

“Function calling” and “tool use” are the same thing. You hand the model a list of tool definitions; the model decides whether to call one, and returns a structured request to call it (name + JSON arguments). The model never executes anything — your code runs the tool and feeds the result back. Function calling is itself a structured-output problem: the arguments are constrained to the tool’s input_schema (and strict: true makes that guarantee exact).

A tool definition is name + description + a JSON Schema for inputs:

tools = [{
    "name": "get_weather",
    "description": "Get current weather for a location. "
                   "Call this when the user asks about current conditions.",
    "strict": True,                       # guarantee args validate exactly
    "input_schema": {
        "type": "object",
        "properties": {
            "location": {"type": "string", "description": "City and state, e.g. Pune, MH"},
            "unit": {"type": "string", "enum": ["celsius", "fahrenheit"]},
        },
        "required": ["location"],
        "additionalProperties": False,
    },
}]

The description is load-bearing

The model routes on the description, not just the name. Be prescriptive about when to call — “Call this when the user asks about current prices or recent events” beats “Gets prices.” Recent frontier models reach for tools more conservatively, so an explicit trigger condition in the description measurably raises the should-call rate. Keep the tool set focused; too many tools dilutes routing.

`tool_choice` — controlling whether the model calls

Value	Behavior
`auto`	model decides whether to use a tool (default)
`any`	model must use at least one tool (no plain-text answer)
`tool` (named)	force a specific tool — e.g. `{"type": "tool", "name": "get_weather"}`
`none`	model may not use tools this turn

any and named-tool are how you turn function calling into a structured-extraction guarantee: force the tool, and the model is obliged to emit arguments matching its schema. Add disable_parallel_tool_use: true to cap it at one call per turn.

6. Parallel tool calls

One assistant turn may contain multiple tool-call requests when the tasks are independent (e.g. weather in three cities, or read-profile + check-inventory at once). Default behavior on modern models is parallel-on.

Map results back by id, and return them all in ONE message

Each tool-call block carries a unique id (tool_use_id). When you return results, every result must carry the matching id, and all results go in a single follow-up message — don’t split them across turns.

# Assistant returned several tool_use blocks; execute and collect ALL results
tool_results = []
for block in response.content:
    if block.type == "tool_use":
        out = run_tool(block.name, block.input)      # your dispatch
        tool_results.append({
            "type": "tool_result",
            "tool_use_id": block.id,                  # ← match by id
            "content": out,
        })
# All results in ONE user message:
messages.append({"role": "user", "content": tool_results})

Splitting results trains the model to stop parallelizing

If you return parallel results in separate messages, the model learns the calls weren’t really concurrent and reverts to serial calling on later turns. One assistant turn of tool_use blocks → one user turn of tool_result blocks. For a tool that failed, still return its result with is_error: true — don’t silently drop it, or the id goes unanswered and the request 400s.

7. The agentic tool-use loop

Structured output is a single call; an agent is a loop. Keep calling the model, executing whatever tools it asks for, feeding results back, until it stops asking.

messages = [{"role": "user", "content": user_input}]
MAX_ITERS = 10
for _ in range(MAX_ITERS):                     # ← hard cap: never `while True`
    resp = client.messages.create(model="claude-opus-4-8", max_tokens=4096,
                                  tools=tools, messages=messages)
 
    if resp.stop_reason == "end_turn":         # model is done
        break
    if resp.stop_reason == "pause_turn":        # server-tool loop paused — resend to resume
        messages.append({"role": "assistant", "content": resp.content})
        continue
 
    # stop_reason == "tool_use": run the tools, append results, loop
    messages.append({"role": "assistant", "content": resp.content})
    results = [
        {"type": "tool_result", "tool_use_id": b.id, "content": run_tool(b.name, b.input)}
        for b in resp.content if b.type == "tool_use"
    ]
    messages.append({"role": "user", "content": results})
else:
    raise RuntimeError("agent exceeded max iterations")   # termination guard fired

Always bound the loop

Termination is the #1 production agent footgun. A bug, a tool that always errors, or two tools that ping-pong will spin forever, burning tokens and latency. Hard rules: a max-iterations cap, a token/cost budget, and a check that you’re making progress. Break on the model’s terminal stop_reason (end_turn) — but also bail on your own ceiling. Tie spend to 15 - Cost, Latency & Performance.

SDK tool runners

Most SDKs ship a “tool runner” that drives this loop for you (decorate a Python function, define a Zod tool in TS) — it calls the API, runs your function, feeds the result back, and stops when the model is done. Use it for the happy path; write the manual loop when you need approval gates, custom logging, or conditional execution.

8. Structured extraction patterns

The most common production use of all this: pull typed records out of unstructured text (resumes, invoices, emails, support tickets).

Pattern	How	Use when
Schema-direct extraction	Structured Outputs with the target schema as the response format	one clean record per document
Extract-via-forced-tool	define an `extract(record)` tool, `tool_choice` = that tool	you also want the model to decide when a record is present
List extraction	schema with a top-level array of records	many records per document (line items, attendees)
Two-pass: reason then extract	free-form analysis call → second call constrained to schema	extraction needs reasoning the strict pass would suppress (see §9)

For all of these: pair the strict schema (shape) with a full validator (values), and decide your repair policy — reject, auto-repair, or re-ask with the validation error.

9. Caveats & risk mitigation

Schema = shape, not correctness (§1, §3). The single most important caveat. Validate argument values server-side, confirm destructive actions before executing, and treat every model-supplied value as untrusted input.
The reasoning-degradation finding (contested). Several 2024–2025 studies reported that forcing strict structured output can reduce reasoning quality ~10–15% versus free-form on some tasks — the format constraint crowds out the model’s natural “think then answer” flow. This is contested: other work finds little or no gap with well-designed schemas, and it’s task- and model-dependent. Treat it as a risk to test for, not a law.
Mitigation: give reasoning a place to live
- Put a reasoning (or scratchpad) field FIRST in the schema, before the answer fields — the model writes its working into the JSON itself, in generation order, before committing to outputs.
- Or decouple: do a free-form chain-of-thought call first, then a second pass that only formats the conclusion into the schema (the §8 two-pass pattern).
- On reasoning models, prefer the model’s native thinking over manual CoT — see 02 - Decoding, Sampling & Reasoning Controls and 03 - Prompt & Context Engineering.
OpenAI strict subset rules (§3). Root object, all-properties-required, additionalProperties: false, no pattern/format/numeric constraints. Enforce real value constraints in the post-validation layer, not the schema.
Handle refusal and truncation. A safety refusal (stop_reason: "refusal" / a refusal field) means the output won’t match your schema — check for it before parsing, or you’ll throw on content[0]. Likewise stop_reason: "max_tokens" means truncated, invalid JSON — raise max_tokens and retry.
First-request schema-compile latency. A new schema incurs a one-time grammar-compilation cost (provider-side cache, ~24h on OpenAI/Anthropic). Complex schemas compile slower (acute with Outlines). Keep schemas stable to stay on the cache.
Penalties corrupt structure. frequency_penalty/presence_penalty above ~0 damage JSON and code (they punish the repeated braces/quotes/keywords structure needs). Keep them at 0 for structured output — see 02 - Decoding, Sampling & Reasoning Controls.
Parallel-call result mapping (§6). Wrong id, split messages, or a dropped failed-tool result will 400 or silently degrade parallelism.
Evaluate the contract. Schema-validity isn’t an eval of usefulness. Track parse rate, value-validation pass rate, and downstream task success separately — see 13 - Evaluation & Evals.

10. MCP vs function calling — don’t conflate them

A frequent interview trap. They operate at different layers:

	Function calling	MCP (Model Context Protocol)
What it is	a model capability — the model emits a request to call a tool	a transport / integration standard — how tools, data, and prompts are served to a client
Layer	inside the model API call	a protocol between an MCP client (your app) and MCP servers
Defines	the `tool_use` request format and arguments	discovery, schemas, and invocation of tools over a wire protocol
Relationship	the mechanism that fires a tool	a standardized way to supply tools that the model then calls via function calling

In short: function calling is how the model asks to use a tool; MCP is a standard way to plug tools in. An MCP-connected tool is still invoked through the model’s function-calling machinery. Full treatment in 05 - Model Context Protocol (MCP).

11. Library landscape (2026)

Library	Layer	What it gives you	Approach
Pydantic (Py) / Zod (TS)	data modeling	native types → JSON Schema + full value validation	the validator behind everything else
Instructor	client wrapper, any provider	Pydantic-typed responses with automatic retry-on-validation-error	validate-and-retry (full constraints)
Outlines	open-weight inference	FSM/regex/JSON-schema constrained generation	constrained decoding (during generation)
XGrammar	open-weight inference	fast grammar engine; default backend for vLLM/SGLang/TensorRT-LLM	constrained decoding (during generation)
BAML	DSL + codegen	define functions/schemas in a typed DSL; generates clients; tolerant “SAP” parser for near-JSON	schema-first; robust parsing over strict masking

Mental model

Pydantic/Zod = the contract + value validator (use everywhere). Provider Structured Outputs = constrained decoding you don’t host. Outlines/XGrammar = constrained decoding you do host. Instructor = retry-based portability across any API. BAML = a schema-first DSL that leans on a forgiving parser instead of token masking.

Sanket's Tutorials

Explorer

04 - Structured Output & Tool Calling

1. The concept: turning free text into a contract

2. JSON mode vs Structured Outputs

3. JSON Schema 2020-12 as the contract — and the provider subset

4. Constrained / grammar-guided decoding (the strongest rung)

Two philosophies, one outcome

5. Tool / function calling: the spec

`tool_choice` — controlling whether the model calls

6. Parallel tool calls

7. The agentic tool-use loop

8. Structured extraction patterns

9. Caveats & risk mitigation

10. MCP vs function calling — don’t conflate them

11. Library landscape (2026)

12. Sources

Table of Contents

Backlinks

Graph View

Sanket's Tutorials

Explorer

04 - Structured Output & Tool Calling

1. The concept: turning free text into a contract

2. JSON mode vs Structured Outputs

3. JSON Schema 2020-12 as the contract — and the provider subset

4. Constrained / grammar-guided decoding (the strongest rung)

Two philosophies, one outcome

5. Tool / function calling: the spec

tool_choice — controlling whether the model calls

6. Parallel tool calls

7. The agentic tool-use loop

8. Structured extraction patterns

9. Caveats & risk mitigation

10. MCP vs function calling — don’t conflate them

11. Library landscape (2026)

12. Sources

Table of Contents

Backlinks

Graph View

`tool_choice` — controlling whether the model calls