LangSmith Playground: Prompt Iteration Without Code

Problem: Prompt Tweaks Shouldn't Require a Deploy Cycle

You change one word in a system prompt. Now you have to edit the file, restart the server, trigger a run, and check logs just to see if it helped. With a non-trivial LLM app, that cycle eats 5–10 minutes per iteration.

LangSmith Playground breaks that loop. It lets you edit prompts, swap models, and compare outputs directly in the browser — no code, no redeploy.

You'll learn:

• How to open any traced run in the Playground with one click
• How to run A/B comparisons across prompt variants
• How to save winning prompts back to your prompt hub

Time: 12 min | Difficulty: Beginner

Why the Playground Exists

Most prompt engineering happens in notebooks or ad-hoc scripts. The problem: you're testing against one input at a time, and history disappears when you close the session.

LangSmith Playground is built on top of the tracing infrastructure you already have. Every run your app makes is capturable. The Playground lets you take any real production run — with real inputs — and replay it against an edited prompt or a different model. You're not making up test cases; you're iterating on what actually failed.

Solution

Step 1: Instrument Your App (Skip If Already Traced)

If your app already sends traces to LangSmith, jump to Step 2. If not, add tracing in two minutes:

pip install langsmith langchain-openai

import os

# LangSmith picks these up automatically
os.environ["LANGCHAIN_TRACING_V2"] = "true"
os.environ["LANGCHAIN_API_KEY"] = "lsv2_your_key_here"
os.environ["LANGCHAIN_PROJECT"] = "my-app"

Run your app once. You'll see the run appear in your LangSmith project dashboard.

Step 2: Open a Run in the Playground

In your LangSmith project, click any LLM run in the trace list. Look for the "Open in Playground" button in the top-right of the run detail panel.

This loads the exact inputs — system prompt, user message, model, and parameters — that produced that run. Nothing is fabricated. You're working with a real failure or a real success.

What you'll see in Playground:

• System prompt (editable)
• Human message (editable)
• Model selector
• Temperature, max tokens, and other sampler controls
• Output panel on the right

Step 3: Edit the Prompt and Re-Run

Click directly into the system prompt field and change it. Then hit Run.

# Before
You are a helpful assistant. Answer the user's question.

# After — more specific, constrains output format
You are a technical support assistant for a SaaS product.
Answer in 3 sentences or fewer. If you don't know, say so directly.
Do not apologize before answering.

The output renders immediately on the right. You can keep editing and re-running in place — each run is logged automatically so you don't lose any version.

Step 4: Compare Variants Side by Side

The most useful Playground feature is the "+ Add variant" button. Click it and you get a second column with its own prompt, model, and parameters.

Use this to test:

• Two different system prompts against the same input
• The same prompt on GPT-4o vs Claude 3.5 Sonnet
• Temperature 0.2 vs 0.8 on a creative writing task

Variant A                          Variant B
─────────────────────────────────  ─────────────────────────────────
System: "Answer in 3 sentences"    System: "Answer in bullet points"
Model:  gpt-4o                     Model:  gpt-4o
Temp:   0.3                        Temp:   0.3

Output: [response A]               Output: [response B]

Both run in parallel. You read both outputs and pick the winner. No code, no logging, no diff tools.

Step 5: Test Against Multiple Inputs

One input isn't enough to trust a prompt change. Use "Run over dataset" to fire the variant against a saved dataset in one click.

If you don't have a dataset yet, create a minimal one:

1. In LangSmith sidebar → Datasets & Testing → New Dataset
2. Add 5–10 representative inputs (copy them from your trace history)
3. Back in Playground → click "Run over dataset" → select your dataset

LangSmith runs your prompt variant against every example and shows pass/fail next to each output. You can define an evaluator (exact match, LLM-as-judge, regex) or just eyeball it.

Step 6: Save the Winning Prompt to the Hub

Once a variant wins, don't copy-paste it back into your codebase manually. Push it to the LangSmith Prompt Hub:

1. Click "Save as prompt" in the Playground toolbar
2. Name it: support-agent-system-v2
3. Add a commit message: Constrain to 3 sentences, remove apology prefix

Pull it in your app code:

from langsmith import Client

client = Client()
prompt = client.pull_prompt("support-agent-system-v2")

# Use directly with LangChain
chain = prompt | model

Now your app always uses the named, versioned prompt. Next iteration: edit in Playground, save a new version, bump the pull call.

Verification

Open your LangSmith project and check Prompts in the sidebar. You should see support-agent-system-v2 listed with its commit history.

Run this to confirm the pull works:

from langsmith import Client

client = Client()
prompt = client.pull_prompt("support-agent-system-v2")
print(prompt.messages[0].prompt.template)

You should see: Your updated system prompt text printed to stdout.

What You Learned

• Playground loads real production runs — no synthetic test data needed
• Side-by-side variant comparison removes guesswork from prompt decisions
• Dataset runs give you statistical signal across more than one input before shipping
• Prompt Hub creates a versioned history that your codebase can pull by name

Limitation: Playground works best for single-turn or two-turn chains. For complex multi-step agents, you'll still need code-level tracing to debug intermediate steps — Playground handles the prompt layer, not the orchestration layer.

Tested on LangSmith SDK 0.3.x, Python 3.12, with GPT-4o and Claude 3.5 Sonnet