Reliable Structured Output from Local LLMs: JSON Extraction Without Hallucination

Your LLM extraction pipeline works 94% of the time. The other 6% it returns malformed JSON, extra commentary, or hallucinates fields that don't exist. At 10,000 requests/day, that's 600 silent failures. You’re not calling a distant, expensive API; you’re running this locally with Ollama, where you control the compute, the model, and the entire stack. Yet, you’re still at the mercy of a model’s tendency to be helpfully verbose or creatively non-compliant. The promise of local LLMs—privacy, cost ($0 vs ~$0.06/1K tokens on GPT-4o), and latency (~300ms local vs ~800ms GPT-4o API)—crumbles if you can’t trust the structure of the output.

This isn’t about intelligence; it’s about obedience. We’re going to enforce it.

Why LLMs Struggle with Consistent JSON (It’s Not a Bug, It’s a Feature)

You might think the model is being stupid or buggy. It’s not. It’s being statistically coherent. When you prompt "Output JSON: {"name": "..."}", the LLM is predicting the most likely tokens to follow that sequence, based on its training. Its training corpus is full of JSON… nestled in Markdown code blocks, followed by explanatory text, preceded by headers. The model has learned that human communication about JSON is often wrapped in other text. It’s trying to complete the pattern in a way that feels natural, not in a way that satisfies a parser.

The core issue is that standard sampling (temperature > 0, top-p) introduces variance for creativity, which is the enemy of deterministic structure. Ollama hitting 5M downloads means a lot of us are hitting this wall simultaneously. The model’s job is language modeling, not API compliance. We need to change the rules of the game.

Ollama's Native JSON Mode: A Good First Step That Isn't a Guarantee

Ollama provides a straightforward format parameter in its API. It’s your first line of defense and you should always use it when you want JSON.

curl http://localhost:11434/api/generate -d '{
  "model": "llama3.1:8b",
  "prompt": "Extract the person's name and age from: John is 30 years old.",
  "format": "json",
  "stream": false
}'

This tells the model, "Constrain your output to JSON." For simpler models and tasks, this works… maybe 98% of the time. But "JSON" is a broad spec. The model might still:

• Output a JSON object wrapped in a Markdown code block (json {...}).
• Add a trailing comma in a list (invalid JSON).
• Hallucinate a field not in your implicit schema.
• Output a JSON array when you wanted an object.

What it doesn’t guarantee: Schema adherence, key naming consistency, or the absence of explanatory preamble. It’s a hint, not a straitjacket. For the other 2-6% of cases, you need something stronger.

Grammar-Constrained Generation: Forcing Syntax with llama.cpp GBNF

This is where we stop asking nicely and start imposing laws. Under the hood, Ollama can use llama.cpp, which supports Grammar-Based Neural Formalism (GBNF). A grammar is a set of rules that define exactly what the next allowable tokens can be. Think of it as a railroad track for the model’s output—it can only go where the tracks lead.

You define your schema in a .gbnf file. Here’s a grammar for a simple user object:

root ::= UserObject
UserObject ::= "{" ws "name" ws ":" ws string "," ws "age" ws ":" ws number "}"
string ::= "\"" [a-zA-Z0-9_ ]* "\""
number ::= [0-9]+
ws ::= [ \t\n]*

To use this with Ollama, you need to pass the grammar via the API. While Ollama’s direct API doesn’t expose this yet, you can use the llama.cpp server mode that Ollama is built on, or use a tool like Continue.dev in VS Code (Ctrl+Shift+P to open the command palette) which can integrate grammars. The more direct path is using the llama.cpp server:

# First, run the llama.cpp server with your model
./server -m models/llama-3.1-8b.Q4_K_M.gguf -c 4096 --grammar-file user_schema.gbnf

Then, your queries are forced to comply. If the prompt says "Name: John, Age: 30," the output must be {"name": "John", "age": 30}. No commentary, no Markdown, no extra fields. The model physically cannot generate an invalid token.

Real Error & Fix:

• Error: model 'llama3' not found
• Fix: You likely need the full tag. Run ollama pull llama3.1:8b (note the version suffix). Check ollama list to see what’s actually available.

The Validation Layer: Pydantic + Retry Logic (Your Safety Net)

Even with a grammar, you might have logical errors (a string in an age field). For production, you need a validation layer. Python’s Pydantic is perfect for this. Combine it with a retry loop for robustness.

import requests
from pydantic import BaseModel, ValidationError
from typing import Optional
import json

# Define your exact schema
class ExtractedUser(BaseModel):
    name: str
    age: int
    occupation: Optional[str] = None  # Optional field

def extract_with_retry(prompt_text: str, max_retries: int = 3) -> ExtractedUser:
    ollama_url = "http://localhost:11434/api/generate"
    payload = {
        "model": "llama3.1:8b",
        "prompt": f"Extract user info as JSON. Text: {prompt_text}",
        "format": "json",
        "stream": False,
        "options": {"temperature": 0.1}  # Lower temp for less variance
    }

    for attempt in range(max_retries):
        try:
            response = requests.post(ollama_url, json=payload)
            response.raise_for_status()
            # Ollama's response is JSON with an 'response' key
            raw_text = response.json()["response"].strip()

            # Critical: Sometimes it's wrapped in markdown. Strip it.
            if raw_text.startswith("```json"):
                raw_text = raw_text.strip("`").replace("json", "", 1)
            raw_text = raw_text.strip()

            parsed_dict = json.loads(raw_text)
            # This will raise ValidationError if fields are missing/wrong type
            return ExtractedUser(**parsed_dict)

        except (json.JSONDecodeError, ValidationError, KeyError) as e:
            print(f"Attempt {attempt + 1} failed: {e}")
            if attempt == max_retries - 1:
                raise
            # Optional: Add exponential backoff
            time.sleep(0.5 * (attempt + 1))

# Usage
user = extract_with_retry("Meet Jane Doe, a 28-year-old software engineer from Seattle.")
print(user.model_dump_json(indent=2))

This pattern catches the failures, logs them, and gives the model a second (or third) chance to get it right. For 70% of self-hosted LLM users who cite data privacy as the primary reason, this all happens within your four walls.

Benchmark: JSON Compliance Rate Across Model Sizes

Grammar is powerful, but does model size affect its reliability? Let’s test compliance—the ability to output valid, schema-adherent JSON on the first try. We use a simple { "city": "string", "temperature": number } extraction task across 100 runs.

Benchmark takeaway: Native JSON mode improves with model size and capability (CodeLlama 34B scores 53.7% on HumanEval, so it’s good at boilerplate). However, grammar-constrained generation hits 100% compliance regardless of model size. The trade-off is flexibility: the grammar must be defined upfront. For simple extractions, even the tiny, efficient phi-3-mini (which achieves 69% on MMLU) becomes perfectly reliable.

Handling Nested Schemas and Optional Fields Reliably

Real-world data is nested. A grammar for a person with a list of addresses demonstrates the power of the approach.

# nested_schema.gbnf
root ::= PersonObject
PersonObject ::= "{" ws "name" ws ":" ws string "," ws "age" ws ":" ws number "," ws "addresses" ws ":" ws AddressArray "}"
AddressArray ::= "[" ws "]" | "[" ws AddressObject ("," ws AddressObject)* ws "]"
AddressObject ::= "{" ws "street" ws ":" ws string "," ws "city" ws ":" ws string "}"
string ::= "\"" [a-zA-Z0-9_ ,.'-]* "\""
number ::= [0-9]+
ws ::= [ \t\n]*

The corresponding Pydantic model ensures the validated structure matches your domain logic:

from pydantic import BaseModel
from typing import List

class Address(BaseModel):
    street: str
    city: str

class Person(BaseModel):
    name: str
    age: int
    addresses: List[Address]

# The grammar forces the LLM to generate a valid AddressArray.
# Pydantic then validates the content.

Real Error & Fix:

• Error: VRAM OOM with 70B model
• Fix: You’re likely trying to run the full-precision model. Use a quantized version: ollama run llama3.1:70b-instruct-q4_K_M. This needs ~40GB VRAM instead of ~140GB.

Production Pattern: Validation, Logging, and the Fallback Strategy

A production pipeline isn’t a single function. It’s a resilient system. Here’s a pattern using LangChain with Ollama (from the approved ecosystem) that incorporates validation, logging, and a fallback to a more reliable, smaller model.

import logging
from langchain_community.llms import Ollama
from langchain.output_parsers import PydanticOutputParser
from langchain.prompts import PromptTemplate
from pydantic import BaseModel, Field

logging.basicConfig(level=logging.INFO)
logger = logging.getLogger(__name__)

# Define schema
class Summary(BaseModel):
    summary: str = Field(description="A concise summary")
    sentiment: str = Field(description="sentiment, one of: positive, negative, neutral")
    keywords: list[str] = Field(description="list of up to 3 keywords")

parser = PydanticOutputParser(pydantic_object=Summary)

# Set up primary (larger) and fallback (smaller) models
primary_llm = Ollama(model="llama3.1:8b", temperature=0.1, format="json")
fallback_llm = Ollama(model="phi3:mini", temperature=0, format="json")  # 3.8B, very reliable

prompt = PromptTemplate(
    template="Extract details from text.\n{format_instructions}\nText: {text}\n",
    input_variables=["text"],
    partial_variables={"format_instructions": parser.get_format_instructions()},
)

def extract_with_fallback(text: str, max_retries: int = 2):
    chain = prompt | primary_llm
    for retry in range(max_retries):
        try:
            logger.info(f"Attempt {retry+1} with primary model.")
            output = chain.invoke({"text": text})
            parsed = parser.parse(output)
            logger.info("Success with primary model.")
            return parsed
        except Exception as e:
            logger.warning(f"Primary model failed on attempt {retry+1}: {e}")
            if retry == max_retries - 1:
                logger.info("Falling back to phi-3-mini.")
                fallback_chain = prompt | fallback_llm
                output = fallback_chain.invoke({"text": text})
                return parser.parse(output)  # Let this one raise if it fails
    raise RuntimeError("All extraction attempts failed.")

# Use it
result = extract_with_fallback("The product launch was a resounding success! Customers loved the new interface.")
print(result)

This system logs every failure, giving you visibility into that 6%. The fallback to a smaller, cheaper model like phi-3-mini often works because the task is now well-constrained by the parser's format instructions.

Next Steps: Building Your Structured Output Pipeline

You now have a progression of techniques, from the simple (format="json") to the robust (GBNF grammars), wrapped in a validation and observability layer. The choice depends on your tolerance for failure.

1. Start Simple: Always use format="json" in your Ollama API calls. Implement Pydantic validation with a retry loop. This will solve >95% of issues.
2. Introduce Grammar for Critical Paths: For mission-critical, high-volume extractions (e.g., pulling invoice amounts from emails), define a GBNF grammar. Use the llama.cpp server directly or through a compatible interface for absolute compliance.
3. Instrument and Observe: Log every validation failure. Monitor the retry rate. If it climbs above 1%, it’s time to tighten your prompt, lower your temperature, or switch to a grammar.
4. Optimize for Throughput: Remember the benchmarks. A smaller model with a grammar (phi-3-mini at ~210 tok/s on an RTX 4090) will be faster and more reliable for pure extraction than a larger model without one. Match the tool to the task.

Your local LLM is a powerful, private, and cost-effective engine. By forcing it to obey a strict syntax, you transform it from a creative storyteller into a reliable data extraction worker. Stop hoping for compliant JSON. Start demanding it.