Prompt Optimization with DSPy

Model merging showed one way to adapt a model without full retraining: combine existing checkpoints, evaluate the candidate, and deploy only after it clears release gates. DSPy applies the same discipline to prompts. Instead of hand-editing instructions until they feel right, you define a prompt program, measure it against examples, compile it offline, and release only a tested artifact.

Programmatic prompt optimization treats prompts as tunable programs evaluated against examples. This chapter introduces DSPy modules, metrics, and optimizers so candidate prompt artifacts are selected through measured iteration instead of hand editing alone.

Imagine trying to write a support workflow for delayed shipments. You could list every step in extreme detail, but if the carrier API, policy wording, or model changes, the prompt might fail. This is what manual prompting feels like after the last few chapters: you've learned to craft chain-of-thought prompts and sprinkle in few-shot examples, but those techniques still leave you hand-tuning strings for every new model or data shift.

DSPy (Declarative Self-improving Python)^[1] changes the workflow. Instead of hand-tuning every instruction, you provide examples of good order-resolution behavior and let an optimizer search for prompts and demonstrations that score well on a declared metric. That optimization can find a better candidate on development data; a separate holdout test and production monitoring still determine whether it is worth deploying.

The problem with manual prompting

Why manual prompts fail at scale

1. Model coupling: A prompt tuned for one model family can degrade badly on another.
2. Fragility: Small wording changes can materially shift quality.
3. Weak iteration loop: How do you systematically compare a revised instruction with the current baseline?
4. Pipeline complexity: When a pipeline chains multiple LM calls, each prompt change can affect downstream behavior.
5. Untracked deployment state: Which prompt, model, dataset, and metric produced last Tuesday's release?

DSPy's thesis

The compiler analogy is useful with a limit: a DSPy program defines a relatively stable interface and control flow, while compilation searches prompt instructions and demonstrations for one configured LM and metric. Unlike a software compiler, DSPy compilation is empirical optimization. It can overfit its data and does not prove correctness.

Instead of writing prompts, write programs. A program defines:

• What you want the LLM to do (signatures)
• How to compose LLM calls (modules)
• What good looks like (metrics)

Then let an optimizer propose prompt instructions and few-shot examples for your chosen model and metric, and compare the compiled candidate with a baseline on held-out data.

Core concepts

Signatures: declarative Input/Output specs

A signature defines the task without specifying how to prompt. It abstracts away the string formatting, acting as a declarative type signature for an LLM call. Think of it like a function signature in Python: you declare the inputs and outputs, but you don't write the body.

The following runnable code shows both inline signatures for quick experiments and class-based typed signatures for production. We'll use a customer support example throughout this article so you can see how the pieces fit together:

The practical shift is that you stop writing the full prompt text by hand. DSPy derives the initial prompt scaffolding from the signature docstring and field descriptions, then optimizers can rewrite instructions and demos during compilation.

Modules: composable LLM building blocks

Modules compose signatures into larger components. They wrap signatures, hold state such as demonstrations, and define execution flow. This separates program structure from compiled prompt state. If you swap a language model or change a signature, keep the structure where appropriate, then recompile and evaluate rather than assuming the previous artifact transfers.

Here's the support pipeline we'll build. It takes a customer ticket plus policy context and generates a resolution with a Chain-of-Thought signature. In production, a retrieval system can fill policy_context; for the core optimization example, keeping context explicit makes the program easier to test. Notice how the logic flows like standard Python code:

Notice that GenerateResolution only declares resolution. dspy.ChainOfThought(GenerateResolution) prepends a reasoning field automatically, so you keep the base signature focused on the task interface and let the module add the reasoning trace.

Metrics: defining "Good"

Optimization requires a clear signal to know if changes improve the pipeline. Metrics evaluate the system's output by returning a numerical score (often 0.0 to 1.0). Unlike traditional machine learning where metrics compare floats, LLM metrics often require semantic evaluation to judge text quality.

A metric function can be as simple as an exact string match or as complex as a separate LLM acting as an automated judge. Because the optimizer runs the metric hundreds of times, it must strike a balance between accuracy and computational cost.

The code below shows a metric for our support pipeline. It combines a strict exact-match shortcut for tiny tests with a simple policy-faithfulness check that verifies the answer cites information from the provided context. Both arguments are ordinary DSPy objects: an example from your dataset and a prediction from your program.

For open-ended support resolutions, you may add an LLM-as-a-judge metric when deterministic checks are too narrow. Judge metrics can vary across runs and can prefer verbose answers or be influenced by output ordering.^[2] Lower-temperature decoding may reduce sampling variance but does not remove systematic bias. Before optimizing against a judge, calibrate it on held-out human-labeled examples, document an acceptance threshold appropriate to the task, and keep deterministic policy checks for critical requirements.

Comparison: Traditional Prompting vs. DSPy

Building a pipeline from scratch

Moving from manual prompting to programmatic optimization is best understood by walking through one concrete pipeline. Let's build the support ticket resolver we sketched above, from signature to compiled artifact.

Define what you want

Start by declaring what the model should do, not how. Our GenerateResolution signature says: "Give me a ticket and some policy context, and I'll produce a resolution." If you want explicit reasoning, choose a module like dspy.ChainOfThought and let it extend the signature for you. Don't write any prompt text yet.

Wire the modules

Combine DSPy modules to create your pipeline. Chain together ChainOfThought, Predict, retrieval modules, or ordinary Python control flow like stacking layers in PyTorch. The structure reflects your logic, not your prompt strings. Our SupportPipeline receives policy passages from the dataset or retrieval layer, then generates a resolution.

Write the metric

Write a function that scores your pipeline's output quality. It should return a number between 0 and 1. Good metrics measure the behavior you care about (accuracy, faithfulness, style alignment), not only string similarity. For our support pipeline, we might combine resolution_accuracy and policy_faithfulness into a single composite score.

Gather examples

Start with representative examples that cover important support slices: late deliveries, refunds, damaged items, and adversarial requests. Sample count is an experimental question; expand or rebalance the set when slice metrics show instability or missing coverage.

Keep separate train, validation, and test splits. Use training examples for bootstrapping, validation examples for selecting a compiled candidate, and test examples only for the promotion decision. MIPROv2 can construct a validation split when you omit one, but an explicit split makes leakage and optimization budget easier to audit.

Compile

Run an optimizer such as BootstrapFewShot or MIPROv2 on your development machine or training instance. It searches instruction and demonstration candidates, then returns the best candidate it observed under its selection metric. That candidate may tie or lose to the uncompiled baseline on a held-out test set, so measure both before deployment. Compilation consumes LM calls; rerun it when the model, metric, adapter, or task distribution changes.

Deploy

Save either the compiled program state to JSON or the full program as a serialized artifact. In production, load the right form and run inference. There's no optimizer loop at runtime; serving uses the optimized prompts and demos.

How the optimizer works

The optimizer is the engine that searches alternative program state. Older DSPy papers and internals often call this a teleprompter, but current docs center the term optimizer. It takes your program, data, and metric, and outputs a compiled candidate with selected parameters such as instructions and demonstrations.

Think of it like a hyperparameter search for prompts. Suppose you have 20 support tickets in your validation set. The optimizer might try:

• Version A: zero-shot with a generic instruction
• Version B: three few-shot examples from successful traces
• Version C: a rewritten instruction plus five optimized demos

It scores candidates on validation tickets and keeps the version with the highest observed validation score. A final test comparison protects against selecting a prompt that happened to fit validation examples unusually well.

Formally, we want to find the parameter set $\phi$ that maximizes the expected metric $M$ over a dataset $D$:

$$\phi^* = \arg\max_{\phi} \mathbb{E}_{x \in D} [M(x, \text{Program}_\phi(x))]$$

Where $\phi$ represents program parameters (instructions and demonstrations), $D$ is typically a held-out validation set, and $M$ is the metric we optimize (for example, accuracy or F1 score, which balances precision and recall).

This is a discrete optimization problem, not gradient descent. The parameters $\phi$ include natural-language instruction strings and sets of few-shot examples, neither of which expose a gradient through a hosted LM call. DSPy optimizers use methods such as random search, surrogate-guided search, or reflective evolution depending on the optimizer selected. Cost depends on candidate count, evaluation set, metric, and configured LMs.

Reading the formula

Try different instruction and demonstration configurations $\phi$, run the resulting program on examples $x$ from the evaluation set $D$, score each output with $M$, and pick $\phi^*$, the configuration with the highest average score. In practice, the expectation $\mathbb{E}$ is estimated with the sample mean over the evaluation set.

The following architecture diagram shows the optimization path from program and examples to a compiled artifact. The search loop happens inside the optimizer; the important boundary is that candidates are scored on held-out examples before one is selected.

Mechanism: BootstrapFewShot

This optimizer works by "teaching" the student model using a teacher model (often the same model).

1. Bootstrapping: The optimizer runs the unoptimized program on the training set.
2. Filtering: It checks which outputs satisfy the metric.
3. Trace Generation: For successful outputs, it captures the trace (inputs, intermediate steps, outputs).
4. Selection: It keeps up to the configured number of metric-accepted traces as demonstrations in the compiled candidate.

If successful traces exist, BootstrapFewShot can place accepted traces into a candidate program's demonstrations. Compare the compiled result with the uncompiled program; bootstrapping is not itself a quality guarantee. This runnable snippet prepares the optimizer without making an LM request. The commented compile call is the boundary that requires a configured LM:

Mechanism: MIPROv2

MIPROv2 is DSPy's documented optimizer for jointly searching instruction text and few-shot examples. It uses grounded instruction proposal plus surrogate-model-guided search over candidate prompt configurations.^[3]

1. Proposal: An LLM generates multiple candidate instructions for the task based on the data and program structure.
2. Search: The optimizer explores the space of (Instruction, Few-Shot Set) pairs.
3. Evaluation: It evaluates candidates on a validation set, optionally with minibatches, to select a candidate for independent testing.

The following runnable example exercises the manual-control guard without performing optimization. In DSPy 3.0.3, if you provide num_candidates yourself, set auto=None and pass num_trials to compile(...). MIPROv2 also requires configured prompt and task LMs before a real compile run:

MIPROv2 imports optuna during its compile search path in the currently tested DSPy version. On machines that run this optimizer, install dspy[optuna].

If you leave valset=None, the currently tested implementation constructs one from the provided trainset. That convenience path can change with DSPy releases and cannot replace a declared test set. Pass explicit train, validation, and test splits for comparisons you intend to deploy.

The execution trace: why the graph matters

In traditional prompt engineering, the evaluation function often only sees the final string output of the language model. If a multi-step prompt outputs a correct answer, the system may assume success even if an intermediate query rewrite or reasoning step was poor.

DSPy supports this through execution traces collected during optimization. Trace-aware optimizers can use successful predictor calls for demonstration bootstrapping, while trace-aware metrics can provide intermediate feedback. For user-facing prompt or response inspection, use LM history rather than depending on private trace object shapes.

Trace collection supports program-level debugging and bootstrapping:

1. Granular Attribution: If a multi-stage LM pipeline fails, the trace helps you pinpoint which predictor produced the bad intermediate output.
2. Intermediate Supervision: You can write metrics that inspect intermediate predictions, such as a generated policy search query or reasoning field, rather than only grading the final answer.
3. Trace-based Few-Shot Generation: When BootstrapFewShot finds a successful execution path, it can reuse that successful trace as few-shot supervision for future runs.

That makes LM pipelines more inspectable during optimization, while metrics and held-out tests remain responsible for quality decisions.

Choosing an optimizer

Choosing the right optimizer depends on your data availability, latency constraints, and the acceptable cost of compilation. When beginning a new project, you might not have enough examples to run complex optimizers. Starting with LabeledFewShot allows you to establish a baseline using manually curated examples.

As you collect representative examples, you can compare BootstrapFewShot against your baseline. This optimizer uses your LM and metric to accept useful traces. When joint search over instructions and demos is worth additional compile budget, evaluate MIPROv2.

Remember that the optimization process itself calls the LLM repeatedly, which incurs API costs and takes time. Balance the compilation cost against the expected improvement in inference accuracy.

Current DSPy docs also list GEPA, a reflective prompt-evolution optimizer, and BootstrapFinetune for adapting model weights rather than only prompt parameters. This article stays focused on BootstrapFewShot, BootstrapRS, and MIPROv2 because they expose demo bootstrapping and instruction search directly.

GEPA uses a reflection LM and textual feedback from execution outcomes to propose instruction revisions, while its Pareto selection can retain candidates that perform differently across objectives.^[4] The paper reports strong results against its evaluated reinforcement-learning baselines, but that result does not select an optimizer for a new application. GEPA needs a suitable reflection LM and a feedback-aware metric, so compare its measured gains and compile cost with simpler optimizers.

Saving and loading programs

Current DSPy exposes two save/load patterns, and mixing them up is a common production mistake.

1. State-only save: program.save("./optimized_support.json", save_program=False) writes program state such as signatures, demonstrations, and module settings to a JSON file. You must recreate compatible Python program structure before calling .load(...).
2. Whole-program save: program.save("./optimized_support/", save_program=True) serializes architecture and state together into a directory. You later restore it with dspy.load("./optimized_support/").

State-only JSON is safer and easier to review. Whole-program loading is more convenient, but it uses cloudpickle to serialize the program, so treat the saved directory like executable code and only load trusted artifacts.

For a trusted whole-program artifact, the corresponding API is compiled_support.save("./optimized_support_v3/", save_program=True) followed by dspy.load("./optimized_support_v3/"). Because that path deserializes a cloudpickled program, only load an artifact you trust and version.

Production deployment

Compilation invokes LMs repeatedly and belongs outside the request path. Its cost depends on optimizer settings, models, metric calls, and dataset size. Separate compile time from serving time in production:

1. Compile Time: Run the optimizer offline on a dev box or training instance.
2. Save: Export either state JSON or a whole-program artifact.
3. Run Time: In production, load the artifact and execute the already-optimized program. No optimizer loop runs online. If you load a whole-program artifact, you must opt into pickle loading explicitly.

The workflow diagram below shows the separation between offline compilation and low-latency production execution:

Use the state-only JSON path from the previous section for ordinary deployments. Reserve whole-program serialization for trusted internal artifacts where convenience is worth the cloudpickle loading risk.

Model swapping is a new experiment

A stable DSPy program interface lets you run a new compilation experiment when you swap underlying models without rewriting all orchestration logic. Because instructions and few-shot examples can be model-specific, recompilation is a candidate-generation step, not evidence that a smaller or cheaper LM meets the old release gates.

For example, you might prototype the support pipeline on a stronger hosted model, then recompile the same DSPy program for a smaller open model such as Llama-3.1-8B-Instruct or Qwen2.5-7B-Instruct.

The optimizer can search new few-shot examples and instruction phrasing for the new model. That is prompt-state optimization, not weight distillation or a substitute for evaluating the target LM.

Cross-model calibration

When swapping models, do not assume an old compiled artifact transfers. Instruction phrasing, adapter behavior, demonstrations, and provider revisions can all change measured output.

Maintain a compilation matrix: for each program version and exact target-model revision, store and evaluate its artifact separately. A provider-side model update is a new comparison, even when the model alias is unchanged.

Multi-hop pipeline patterns

For complex support tickets that span multiple policies, DSPy's modularity can represent multi-step retrieval. This sketch requires a configured retriever and LM, so it is architecture pseudocode rather than an offline runnable example:

Each query-generation module can expose its own prompt state to optimization. Whether optimizing multiple hops improves retrieval and final answers requires trace inspection and held-out evaluation; a larger search surface can also spend more compile budget.

When beginners get stuck

These failure modes are worth checking early because each can consume compile budget without producing a deployable artifact.

The "exact match" trap

Symptom: Your metric uses exact string comparison on long-form text, so the optimizer gives up because almost no output ever scores 1.0.

Cause: You're treating generation like classification. A resolution that says "Your package will arrive tomorrow" and one that says "Estimated delivery: tomorrow" mean the same thing, but exact match gives them a 0.

Fix: Use an LLM-as-judge metric, or at least a heuristic like "Does the resolution mention a policy number and a timeframe?" Exact match is fine for short labels (urgency levels), but it's the wrong tool for open-ended generation.

Data hunger

Symptom: You delay compiling because you think you need thousands of labeled tickets.

Cause: You're importing habits from traditional deep learning, where more data usually helps. DSPy optimizers search over prompt configurations, not model weights.

Fix: Start with a small, representative split, measure uncertainty and slice coverage, then add examples where evaluation shows gaps. Do not claim a required sample count before measuring the task.

Not inspecting the compiled prompt

Symptom: The compiled pipeline works on your test set, but fails strangely in production and you can't explain why.

Cause: You treated the optimizer as a black box. You don't know which few-shot examples it selected or how it rewrote the instructions.

Fix: After compilation, inspect prompt history with the documented dspy.inspect_history(n=...) helper or the configured LM's history method. If selected examples are edge cases or instructions are misleading, revise training coverage or the metric and rerun the comparison.

Hidden production mistakes

Some DSPy failures don't look like prompt failures at first:

• Treating DSPy as a prompt management library instead of an optimizer. If you never define a metric and compile against examples, you're using the interface layer but skipping the learning loop.
• Passing num_candidates or num_trials to MIPROv2 while leaving auto enabled. Manual mode requires auto=None, num_candidates on the optimizer, and num_trials during compile(...).
• Installing only the base package on machines that run MIPROv2. The optimizer imports optuna during compilation, so compile hosts should install dspy[optuna].
• Over-optimizing on a tiny validation set. If a candidate wins because it memorized five tickets, it won't survive real support traffic.
• Assuming a JSON save captures your Python program architecture. State-only JSON needs the same module class recreated before .load(...).
• Loading whole-program artifacts from untrusted sources. dspy.load(...) restores a cloudpickled program, so treat it like executable code.
• Ignoring compile cost. Heavy optimizers can call LMs many times, so budget compile runs separately from serving traffic.

Practice checkpoints

Use these questions to check whether you can reason about DSPy beyond syntax.

How does MIPROv2 differ from simple few-shot selection?

MIPROv2 jointly optimizes instruction text and demonstration sets. BootstrapFewShot mainly bootstraps demos from successful traces. MIPROv2 searches a larger space by proposing grounded instructions, pairing them with demo candidates, and using Bayesian optimization over measured candidate programs.^[3]

Why is model coupling a serious issue in traditional prompt engineering?

A prompt can become overfit to one model family. When you move to a new provider, checkpoint, tokenizer, adapter, or decoding setup, the old wording may degrade. DSPy doesn't remove that risk, but it makes the migration controlled: keep the program interface and metric, then recompile for the target model.

What does the trace parameter give a metric?

It can expose intermediate predictor calls rather than only the final answer. That lets a metric grade a generated search query, a reasoning field, or a policy-selection step inside a multi-stage program. Exact trace internals can change across DSPy versions, so production metrics should depend on documented behavior and local tests, not private object shapes.

How are state-only JSON artifacts different from whole-program saves?

program.save("./artifact.json", save_program=False) stores state. You recreate the Python module class, then call .load(...). program.save("./artifact_dir/", save_program=True) stores the program with cloudpickle, then dspy.load("./artifact_dir/") restores it. JSON is safer and easier to review. Whole-program loading is convenient but only belongs in trusted environments.

What you should be able to defend

By the end, you should be able to:

• Explain why a DSPy Signature is an interface, while a prompt template mixes interface and implementation.
• Describe how optimizers select or generate useful few-shot examples.
• Write a custom metric that scores the behavior you care about, not only string similarity.
• Contrast manual prompt editing with programmatic optimization against train and validation examples.
• Explain the production lifecycle: compile offline, version the artifact, serve without an optimizer loop, monitor drift, and recompile when behavior changes.
• Choose state-only JSON or whole-program serialization and justify the security tradeoff.

What to remember

• DSPy treats prompts as parameters to be optimized, not strings to be manually crafted.
• Signatures define intent; Modules define structure; Optimizers fill in the details.
• BootstrapFewShot proposes demonstration-augmented programs from metric-accepted traces; compare its candidate against the baseline.
• Compile offline, serve without an optimizer loop: Optimization happens before deployment. Re-run it when your model, metric, or data distribution changes.
• Program structure can transfer; compiled state might not: When you change models, recompile and compare against release gates instead of blindly reusing old prompt artifacts.