LangSmith

LangSmith is Anthropic's observability and evaluation platform for LLM applications. It gives you full visibility into every LLM call, chain, and agent run — so you can debug failures, measure quality, and ship reliable AI features.

What LangSmith Solves

Without observability, LLM apps are black boxes. You can't tell why a response was bad, which prompt version performs better, or whether your RAG pipeline is actually retrieving the right context. LangSmith makes all of this visible.

• Tracing — every LLM call, retrieval, and tool use logged with inputs/outputs and latency
• Evaluation — run automated quality tests against ground-truth datasets
• Prompt versioning — manage and compare prompt versions like code
• Cost analytics — track token usage and spend per feature, user, or chain
• Production monitoring — alerts when quality degrades

Quick Start

import os
os.environ["LANGCHAIN_TRACING_V2"] = "true"
os.environ["LANGCHAIN_API_KEY"] = "your-api-key"
os.environ["LANGCHAIN_PROJECT"] = "my-app"

# Your existing LangChain code now auto-traces to LangSmith
from langchain_openai import ChatOpenAI
llm = ChatOpenAI(model="gpt-4o")
llm.invoke("What is RAG?")  # This call appears in LangSmith dashboard

Learning Path

1. Setup tracing — env vars, project organization, first trace
2. Read traces — understand the waterfall view, find slow nodes
3. Create a dataset — capture real examples for regression testing
4. Run evaluations — automated LLM-as-judge scoring
5. CI/CD integration — block deploys when quality drops

LangSmith vs Alternatives