Structured Output from LLMs: JSON Mode, Function Calling, Zod Validation

Getting consistent structured output from LLMs is a foundational skill for production AI applications. This guide covers every approach from fragile text parsing to guaranteed JSON schema enforcement.

The problem with unstructured output

// ❌ Fragile — the model may add explanation text, use different casing, etc.
const response = await openai.chat.completions.create({
  messages: [{ role: 'user', content: 'Extract the sentiment. Reply with positive, negative, or neutral.' }],
});
// response.content might be:
// "The sentiment is positive."
// "Positive"
// "positive."
// "Based on my analysis, this text appears to be POSITIVE."
const sentiment = response.choices[0].message.content;  // 🤞 hope it parses

Approach 1: JSON mode

Forces the model to output valid JSON. Does not enforce a specific schema.

const response = await openai.chat.completions.create({
  model: 'gpt-4o-mini',
  messages: [
    {
      role: 'system',
      content: 'You are a data extractor. Always respond with a JSON object.',
    },
    {
      role: 'user',
      content: 'Extract from: "Alice bought 3 apples for $4.50 on June 10."',
    },
  ],
  response_format: { type: 'json_object' },
});

const data = JSON.parse(response.choices[0].message.content!);
// { customer: "Alice", product: "apples", quantity: 3, total: 4.50, date: "June 10" }

Limitation: JSON mode guarantees valid JSON but not a specific shape. Use Zod to validate the structure.

Approach 2: JSON schema (strict mode)

Guarantees both valid JSON and exact schema compliance via constrained decoding:

const schema = {
  type: 'object',
  properties: {
    customer:  { type: 'string' },
    product:   { type: 'string' },
    quantity:  { type: 'number' },
    total:     { type: 'number' },
    date:      { type: 'string', description: 'ISO 8601 date' },
    sentiment: { type: 'string', enum: ['positive', 'negative', 'neutral'] },
  },
  required: ['customer', 'product', 'quantity', 'total', 'date', 'sentiment'],
  additionalProperties: false,
};

const response = await openai.chat.completions.create({
  model: 'gpt-4o',
  messages,
  response_format: {
    type: 'json_schema',
    json_schema: {
      name: 'OrderExtraction',
      strict: true,
      schema,
    },
  },
});

const data = JSON.parse(response.choices[0].message.content!);
// Guaranteed to match schema — no Zod needed for structure, but still validate values

Approach 3: Function / tool calling

Use tool calling when you want to trigger application logic, not just parse data:

const tools = [{
  type: 'function',
  function: {
    name: 'save_order',
    description: 'Save an extracted order to the database',
    parameters: {
      type: 'object',
      properties: {
        customer: { type: 'string' },
        items: {
          type: 'array',
          items: {
            type: 'object',
            properties: {
              product:  { type: 'string' },
              quantity: { type: 'number' },
              price:    { type: 'number' },
            },
            required: ['product', 'quantity', 'price'],
            additionalProperties: false,
          },
        },
        total: { type: 'number' },
      },
      required: ['customer', 'items', 'total'],
      additionalProperties: false,
    },
  },
}];

const response = await openai.chat.completions.create({
  model: 'gpt-4o',
  messages,
  tools,
  tool_choice: { type: 'function', function: { name: 'save_order' } },
});

const toolCall = response.choices[0].message.tool_calls![0];
const args = JSON.parse(toolCall.function.arguments);
await saveOrderToDatabase(args);

Runtime validation with Zod

Even with JSON schema mode, validate values with Zod before using them in your application:

import { z } from 'zod';

const OrderSchema = z.object({
  customer:  z.string().min(1),
  product:   z.string().min(1),
  quantity:  z.number().int().positive(),
  total:     z.number().positive(),
  date:      z.string().regex(/^d{4}-d{2}-d{2}$/),
  sentiment: z.enum(['positive', 'negative', 'neutral']),
});

type Order = z.infer<typeof OrderSchema>;

function parseOrder(rawJson: string): Order {
  const parsed = JSON.parse(rawJson);
  return OrderSchema.parse(parsed);  // throws ZodError on validation failure
}

// Usage
try {
  const order = parseOrder(response.choices[0].message.content!);
  await processOrder(order);
} catch (err) {
  if (err instanceof z.ZodError) {
    logger.error('LLM output failed validation', err.errors);
    // Retry with richer context or fallback
  }
}

Zod + OpenAI with zod-to-json-schema

import { z } from 'zod';
import { zodToJsonSchema } from 'zod-to-json-schema';

const SentimentSchema = z.object({
  sentiment:   z.enum(['positive', 'negative', 'neutral']),
  confidence:  z.number().min(0).max(1),
  keyPhrases:  z.array(z.string()),
  summary:     z.string(),
});

const jsonSchema = zodToJsonSchema(SentimentSchema, { name: 'SentimentAnalysis' });

const response = await openai.chat.completions.create({
  model: 'gpt-4o',
  messages,
  response_format: {
    type: 'json_schema',
    json_schema: {
      name: 'SentimentAnalysis',
      strict: true,
      schema: jsonSchema.definitions!['SentimentAnalysis'],
    },
  },
});

const result = SentimentSchema.parse(JSON.parse(response.choices[0].message.content!));

Retry on parse failure

async function reliableExtract<T>(
  prompt: string,
  schema: z.ZodType<T>,
  maxRetries = 3
): Promise<T> {
  const messages: OpenAI.ChatCompletionMessageParam[] = [
    { role: 'user', content: prompt },
  ];

  for (let attempt = 1; attempt <= maxRetries; attempt++) {
    const response = await openai.chat.completions.create({
      model: 'gpt-4o',
      messages,
      response_format: { type: 'json_object' },
    });

    const content = response.choices[0].message.content!;
    messages.push({ role: 'assistant', content });

    try {
      return schema.parse(JSON.parse(content));
    } catch (err) {
      if (attempt === maxRetries) throw err;
      messages.push({
        role: 'user',
        content: `Validation failed: ${err}. Please fix the JSON and try again.`,
      });
    }
  }
  throw new Error('All retries exhausted');
}

Approach comparison

Takeaway

Use JSON schema strict mode as your default for extraction tasks — it eliminates parsing failures at the source. Add Zod validation for business-critical paths where value constraints matter, not just structure.