Prompt Optimization with DSPy

Model merging showed one way to adapt a model without full retraining: combine existing checkpoints, evaluate the candidate, and deploy only after it clears release gates. DSPy applies the same discipline to prompts. Instead of hand-editing instructions until they feel right, you define a prompt program, measure it against examples, compile it offline, and release only a tested artifact.

A developer assistant that answers CI and deploy-policy questions can start with hand-tuned instructions, but test logs, runbook wording, and model revisions keep changing. A prompt that worked last month may regress after any one of those changes.

DSPy^{[1]Reference 1DSPy: Compiling Declarative Language Model Calls into Self-Improving Pipelines.https://arxiv.org/abs/2310.03714} treats prompts as tunable program state. You define modules and a metric, provide examples of good incident-resolution behavior, and let a DSPy optimizer search for instructions and demonstrations that score well on development data. A separate holdout test and production monitoring still determine whether the compiled candidate is worth deploying.

The problem with manual prompting

1. Model coupling: A prompt tuned for one model family can degrade badly on another.
2. Fragility: Small wording changes can materially shift quality.
3. Weak iteration loop: How do you systematically compare a revised instruction with the current baseline?
4. Pipeline complexity: When a pipeline chains multiple LM calls, each prompt change can affect downstream behavior.
5. Untracked deployment state: Which prompt, model, dataset, and metric produced last Tuesday's release?

DSPy's thesis

The compiler analogy is useful with a limit: a DSPy program defines a relatively stable interface and control flow, while compilation searches prompt instructions and demonstrations for one configured LM and metric. Unlike a software compiler, DSPy compilation is empirical optimization. It can overfit its data and doesn't prove correctness.

Instead of writing prompts, write programs. A program defines:

• What you want the LLM to do (signatures)
• How to compose LLM calls (modules)
• What good looks like (metrics)

Then let an optimizer propose prompt instructions and few-shot examples for your chosen model and metric, and compare the compiled candidate with a baseline on held-out data.

Core concepts

Signatures: declarative Input/Output specs

A signature defines the task without specifying how to prompt. It abstracts away the string formatting and acts like a Python function signature for an LLM call: you declare the inputs and outputs, but you don't write the body.

This runnable code uses both inline signatures for quick experiments and class-based typed signatures for production. The examples stay with one deploy-policy assistant so the pieces fit together:

The practical shift is that you stop writing the full prompt text by hand. DSPy derives the initial prompt scaffolding from the signature docstring and field descriptions, then optimizers can rewrite instructions and demos during compilation.

Modules: composable LLM building blocks

Modules compose signatures into larger components. They wrap signatures, hold state such as demonstrations, and define execution flow. This separates program structure from compiled prompt state. If you swap a language model or change a signature, keep the structure where appropriate, then recompile and evaluate rather than assuming the previous artifact transfers.

Build a policy pipeline that takes an engineer's ticket plus policy context and generates a resolution with a Chain-of-Thought signature. In production, a retrieval system can fill policy_context; for the core optimization example, keeping context explicit makes the program easier to test. The logic flows like standard Python code:

Notice that GenerateResolution only declares resolution. dspy.ChainOfThought(GenerateResolution) prepends a reasoning field automatically, so you keep the base signature focused on the task interface and let the module add a generated rationale. That rationale isn't the same thing as the DSPy execution trace introduced later: the trace records predictor calls, inputs, and outputs while the program runs.

Metrics: defining a score

Optimization requires a clear signal to know if changes improve the pipeline. Metrics evaluate the system's output by returning a numerical score (often 0.0 to 1.0). Unlike traditional machine learning where metrics compare floats, LLM metrics often require semantic evaluation to judge text quality.

A metric function can be as simple as an exact string match or as complex as a separate LLM acting as an automated judge. Because an optimizer can run the metric repeatedly, it must strike a balance between accuracy and computational cost.

This example shows a metric for our policy pipeline. It combines a strict exact-match shortcut for tiny tests with a simple policy-faithfulness check that verifies the answer cites information from the provided context. Both arguments are ordinary DSPy objects: an example from your dataset and a prediction from your program.

For open-ended incident resolutions, you may add an LLM-as-a-judge metric when deterministic checks are too narrow. Judge metrics can vary across runs and can prefer verbose answers or be influenced by output ordering.^{[2]Reference 2Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena.https://arxiv.org/abs/2306.05685} Lower-temperature decoding may reduce sampling variance but doesn't remove systematic bias. Before optimizing against a judge, calibrate it on held-out human-labeled examples, document an acceptance threshold appropriate to the task, and keep deterministic policy checks for critical requirements.

Comparison: Traditional Prompting vs. DSPy

Building a pipeline from scratch

Moving from manual prompting to programmatic optimization is best understood through one concrete pipeline. The deploy-policy resolver above can be written from signature to compiled artifact.

Define what you want

Start by declaring what the model should do, not how. Our GenerateResolution signature says: "Give me a ticket and some policy context, and I'll produce a resolution." If you want explicit reasoning, choose a module like dspy.ChainOfThought and let it extend the signature for you. Don't write any prompt text yet.

Wire the modules

Combine DSPy modules to create your pipeline. Chain together ChainOfThought, Predict, retrieval modules, or ordinary Python control flow like stacking layers in PyTorch. The structure reflects your logic, not your prompt strings. Our PolicyPipeline receives policy passages from the dataset or retrieval layer, then generates a resolution.

Write the metric

Write a function that scores your pipeline's output quality. It should return a number between 0 and 1. Good metrics measure the behavior you care about (accuracy, faithfulness, style alignment), not string similarity alone. For our deploy-policy pipeline, we might combine resolution_accuracy and policy_faithfulness into a single composite score.

Gather examples

Start with representative examples that cover important developer-assistant slices: CI failures, rollback approvals, break-glass access, and adversarial requests. Sample count is an experimental question; expand or rebalance the set when slice metrics show instability or missing coverage.

Keep separate train, validation, and test splits. Use training examples for bootstrapping, validation examples for selecting a compiled candidate, and test examples only for the promotion decision. MIPROv2 can construct a validation split when you omit one, but an explicit split makes leakage and optimization budget easier to audit.

Compile

Run an optimizer such as BootstrapFewShot or MIPROv2 on your development machine or training instance. It searches instruction and demonstration candidates, then returns the best candidate it observed under the selection metric. That candidate may tie or lose to the uncompiled baseline on a held-out test set, so measure both before deployment. Compilation consumes LM calls; rerun it when the model, metric, adapter, or task distribution changes.

Deploy

Save either the compiled program state to JSON or the full program as a serialized artifact. In production, load the right form and run inference. There's no optimizer loop at runtime; serving uses the optimized prompts and demos.

How the optimizer works

The optimizer is the engine that searches alternative program state. Older DSPy papers and internals often call this a teleprompter, but current docs center the term optimizer. It takes your program, data, and metric, and outputs a compiled candidate with selected parameters such as instructions and demonstrations.

Prompt optimization behaves like hyperparameter search over prompt programs. Suppose you have 20 deploy-policy tickets in your validation set. The optimizer might try:

• Version A: zero-shot with a generic instruction
• Version B: three few-shot examples from successful traces
• Version C: a rewritten instruction plus five optimized demos

It scores candidates on validation tickets and keeps the version with the highest observed validation score. A final test comparison protects against selecting a prompt that happened to fit validation examples unusually well.

Formally, we want to find the parameter set $\phi$ that maximizes the expected metric $M$ over a dataset $D$:

$$\phi^* = \arg\max_{\phi} \mathbb{E}_{x \in D} [M(x, \text{Program}_\phi(x))]$$

Where $\phi$ represents program parameters (instructions and demonstrations), $D$ is typically a held-out validation set, and $M$ is the metric we optimize (for example, accuracy or F1 score, which balances precision and recall).

This is a discrete optimization problem, not gradient descent. The parameters $\phi$ include natural-language instruction strings and sets of few-shot examples, neither of which expose a gradient through a hosted LM call. DSPy optimizers use methods such as random search, surrogate-guided search, or reflective evolution depending on the optimizer selected. Cost depends on candidate count, evaluation set, metric, and configured LMs.

Reading the formula

Try different instruction and demonstration configurations $\phi$, run the resulting program on examples $x$ from the evaluation set $D$, score each output with $M$, and pick $\phi^*$, the configuration with the highest average score. In practice, the expectation $\mathbb{E}$ is estimated with the sample mean over the evaluation set.

The optimizer search loop runs from program and examples to a compiled artifact. Validation picks a candidate inside the optimizer; holdout and release gates still decide promotion.

Mechanism: BootstrapFewShot

BootstrapFewShot runs a teacher program (often the same unoptimized DSPy program, using the same configured LM) and turns metric-accepted execution traces into demonstrations.

1. Bootstrapping: The optimizer runs the teacher program on the training set.
2. Filtering: It checks which outputs satisfy the metric.
3. Trace Generation: For successful outputs, it captures the trace (inputs, intermediate steps, outputs).
4. Selection: It keeps up to the configured number of metric-accepted traces as demonstrations in the compiled candidate.

If successful traces exist, BootstrapFewShot can place accepted traces into a candidate program's demonstrations. Compare the compiled result with the uncompiled program; bootstrapping isn't itself a quality guarantee. This runnable snippet prepares the optimizer without making an LM request. The commented compile call is the boundary that requires a configured LM:

Mechanism: MIPROv2

MIPROv2 is DSPy's documented optimizer for jointly searching instruction text and few-shot examples. It uses grounded instruction proposal plus surrogate-model-guided search over candidate prompt configurations.^{[3]Reference 3Optimizing Instructions and Demonstrations for Multi-Stage Language Model Programshttps://arxiv.org/abs/2406.11695}

1. Proposal: An LLM generates multiple candidate instructions for the task based on the data and program structure.
2. Search: The optimizer explores the space of (Instruction, Few-Shot Set) pairs.
3. Evaluation: It evaluates candidates on a validation set, optionally with minibatches, to select a candidate for independent testing.

This runnable example exercises the manual-control guard without performing optimization. In MIPROv2 manual mode, if you provide num_candidates yourself, set auto=None and pass num_trials to compile(...). MIPROv2 also requires configured prompt and task LMs before a real compile run:

MIPROv2 uses optuna during its compile search path. On machines that run this optimizer, install dspy[optuna].

If you leave valset=None, the current implementation constructs one from the provided trainset. That convenience path can change with DSPy releases and can't replace a declared test set. Pass explicit train, validation, and test splits for comparisons you intend to deploy.

The execution trace: why the graph matters

In traditional prompt engineering, the evaluation function often only sees the final string output of the language model. If a multi-step prompt outputs a correct answer, the system may assume success even if an intermediate query rewrite or reasoning step was poor.

DSPy supports this through execution traces collected during optimization. Trace-aware optimizers can use successful predictor calls for demonstration bootstrapping, while trace-aware metrics can provide intermediate feedback. For user-facing prompt or response inspection, use LM history rather than depending on private trace object shapes.

Trace collection supports program-level debugging and bootstrapping:

1. Granular Attribution: If a multi-stage LM pipeline fails, the trace identifies which predictor produced the bad intermediate output.
2. Intermediate Supervision: You can write metrics that inspect intermediate predictions, such as a generated policy search query or reasoning field, rather than only grading the final answer.
3. Trace-based Few-Shot Generation: When BootstrapFewShot finds a successful execution path, it can reuse that successful trace as few-shot supervision for future runs.

That makes LM pipelines more inspectable during optimization, while metrics and held-out tests remain responsible for quality decisions.

Choosing an optimizer

Choosing the right optimizer depends on your data availability, latency constraints, and the acceptable cost of compilation. When beginning a new project, you might not have enough examples to run complex optimizers. Starting with LabeledFewShot gives you a baseline from provided labeled examples.

As you collect representative examples, you can compare BootstrapFewShot against your baseline. This optimizer uses your LM and metric to accept useful traces. When joint search over instructions and demos is worth additional compile budget, evaluate MIPROv2.

The optimization process itself calls the LLM repeatedly, which incurs API costs and takes time. Balance the compilation cost against the expected improvement in inference accuracy.

Current DSPy docs also list GEPA, a reflective prompt-evolution optimizer, and BootstrapFinetune for adapting model weights rather than only prompt parameters. The examples stay focused on BootstrapFewShot, BootstrapRS, and MIPROv2 because they expose demo bootstrapping and instruction search directly.

GEPA uses a reflection LM and textual feedback from execution outcomes to propose instruction revisions, while its Pareto selection can retain candidates that perform differently across objectives.^{[4]Reference 4GEPA: Reflective Prompt Evolution Can Outperform Reinforcement Learninghttps://arxiv.org/abs/2507.19457} The paper reports strong results against its evaluated reinforcement-learning baselines, but that result doesn't select an optimizer for a new application. GEPA needs a suitable reflection LM and a feedback-aware metric, so compare its measured gains and compile cost with simpler optimizers.

Saving and loading programs

Current DSPy exposes two save/load patterns, and mixing them up is a common production mistake.

1. State-only save: program.save("./optimized_policy.json", save_program=False) writes program state such as signatures, demonstrations, and module settings to a JSON file. You must recreate compatible Python program structure before calling .load(...).
2. Whole-program save: program.save("./optimized_policy/", save_program=True) serializes architecture and state together into a directory. You later restore a trusted artifact with dspy.load("./optimized_policy/", allow_pickle=True).

State-only JSON is safer and easier to review. Whole-program loading is more convenient, but it uses cloudpickle to serialize the program, so treat the saved directory like executable code and only load trusted artifacts.

For a trusted whole-program artifact, the corresponding API is compiled_policy.save("./optimized_policy_v3/", save_program=True) followed by dspy.load("./optimized_policy_v3/", allow_pickle=True). Because that path deserializes a cloudpickled program, only load an artifact you trust and version.

Production deployment

Compilation invokes LMs repeatedly and belongs outside the request path. Its cost depends on optimizer settings, models, metric calls, and dataset size. Separate compile time from serving time in production:

1. Compile Time: Run the optimizer offline on a dev box or training instance.
2. Save: Export either state JSON or a whole-program artifact.
3. Run Time: In production, load the artifact and execute the already-optimized program. No optimizer loop runs online. If you load a whole-program artifact, you must opt into pickle loading explicitly.

Offline compilation and low-latency production execution stay separate:

Use the state-only JSON path from the previous section for ordinary deployments. Reserve whole-program serialization for trusted internal artifacts where convenience is worth the cloudpickle loading risk.

Model swapping is a new experiment

A stable DSPy program interface lets you run a new compilation experiment when you swap underlying models without rewriting all orchestration logic. Because instructions and few-shot examples can be model-specific, recompilation is a candidate-generation step, not evidence that a smaller or cheaper LM meets the old release gates.

For example, you might prototype the deploy-policy pipeline on a stronger hosted model, then recompile the same DSPy program for a smaller open-weight model that your serving stack can run.

The optimizer can search new few-shot examples and instruction phrasing for the new model. That's prompt-state optimization, not weight distillation or a substitute for evaluating the target LM.

Cross-model calibration

When swapping models, don't assume an old compiled artifact transfers. Instruction phrasing, adapter behavior, demonstrations, and provider revisions can all change measured output.

Maintain a compilation matrix: for each program version and exact target-model revision, store and evaluate its artifact separately. A provider-side model update is a new comparison, even when the model alias is unchanged.

Multi-hop pipeline patterns

For complex deploy-policy tickets that span multiple policies, DSPy's modularity can represent multi-step retrieval. This sketch requires a configured search function and LM, so it's architecture pseudocode rather than an offline runnable example:

Each query-generation module can expose its own prompt state to optimization. Whether optimizing multiple hops improves retrieval and final answers requires trace inspection and held-out evaluation; a larger search surface can also spend more compile budget.

When optimization stalls

Check these failure modes early because each can consume compile budget without producing a deployable artifact.

The "exact match" trap

• Symptom: Your metric uses exact string comparison on long-form text, so the optimizer gives up because almost no output ever scores 1.0.

• Cause: You're treating generation like classification. A resolution that says "Run rollback after owner approval" and one that says "Rollback approved: run it now" may mean the same thing in context, but exact match gives them a 0.

• Fix: Use an LLM-as-judge metric, or at least a heuristic like "Does the resolution mention the relevant policy and required approval?" Exact match is fine for short labels (urgency levels), but it's the wrong tool for open-ended generation.

Data hunger

• Symptom: You delay compiling because you think you need thousands of labeled tickets.

• Cause: You're importing habits from traditional deep learning, where more data usually helps. DSPy optimizers search over prompt configurations, not model weights.

• Fix: Start with a small, representative split, measure uncertainty and slice coverage, then add examples where evaluation shows gaps. Don't claim a required sample count before measuring the task.

Not inspecting the compiled prompt

• Symptom: The compiled pipeline works on your test set, but fails strangely in production and you can't explain why.

• Cause: You didn't inspect the optimizer output. You don't know which few-shot examples it selected or how it rewrote the instructions.

• Fix: After compilation, inspect prompt history with the documented dspy.inspect_history(n=...) helper or the configured LM's history method. If selected examples are edge cases or instructions are misleading, revise training coverage or the metric and rerun the comparison.

Hidden production mistakes

Some DSPy failures don't look like prompt failures at first:

• Treating DSPy as a prompt management library instead of an optimizer. If you never define a metric and compile against examples, you're using the interface layer but skipping the learning loop.
• Passing num_candidates or num_trials to MIPROv2 while leaving auto enabled. Manual mode requires auto=None, num_candidates on the optimizer, and num_trials during compile(...).
• Installing only the base package on machines that run MIPROv2. The optimizer imports optuna during compilation, so compile hosts should install dspy[optuna].
• Over-optimizing on a tiny validation set. If a candidate wins because it memorized five tickets, it won't survive real developer traffic.
• Assuming a JSON save captures your Python program architecture. State-only JSON needs the same module class recreated before .load(...).
• Loading whole-program artifacts from untrusted sources. dspy.load(...) restores a cloudpickled program, so treat it like executable code.
• Ignoring compile cost. Heavy optimizers can call LMs many times, so budget compile runs separately from serving traffic.

Quick recall prompts

Use these questions to check whether you can reason about DSPy beyond syntax.

How does MIPROv2 differ from simple few-shot selection? MIPROv2 jointly optimizes instruction text and demonstration sets. BootstrapFewShot mainly bootstraps demos from successful traces. MIPROv2 searches a larger space by proposing grounded instructions, pairing them with demo candidates, and using surrogate-guided search over measured candidate programs.^{[3]Reference 3Optimizing Instructions and Demonstrations for Multi-Stage Language Model Programshttps://arxiv.org/abs/2406.11695}

Why is model coupling a serious issue in traditional prompt engineering? A prompt can become overfit to one model family. When you move to a new provider, checkpoint, tokenizer, adapter, or decoding setup, the old wording may degrade. DSPy doesn't remove that risk, but it makes the migration controlled: keep the program interface and metric, then recompile for the target model.

What does the trace parameter give a metric? It can expose intermediate predictor calls rather than only the final answer. That lets a metric grade a generated search query, a reasoning field, or a policy-selection step inside a multi-stage program. Exact trace internals can change across DSPy versions, so production metrics should depend on documented behavior and local tests, not private object shapes.

How are state-only JSON artifacts different from whole-program saves? program.save("./artifact.json", save_program=False) stores state. You recreate the Python module class, then call .load(...). program.save("./artifact_dir/", save_program=True) stores the program with cloudpickle, then dspy.load("./artifact_dir/", allow_pickle=True) restores it. JSON is safer and easier to review. Whole-program loading is convenient but only belongs in trusted environments.

Before you continue

Confirm you can:

• Explain why a DSPy Signature is an interface, while a prompt template mixes interface and implementation.
• Describe how optimizers select or generate useful few-shot examples.
• Write a custom metric that scores the behavior you care about, not string similarity alone.
• Contrast manual prompt editing with programmatic optimization against train and validation examples.
• Explain the production lifecycle: compile offline, version the artifact, serve without an optimizer loop, monitor drift, and recompile when behavior changes.
• Choose state-only JSON or whole-program serialization and justify the security tradeoff.

Key takeaways

• DSPy treats prompts as parameters to be optimized, not strings to be manually crafted.
• Signatures define intent; Modules define structure; Optimizers fill in the details.
• BootstrapFewShot proposes demonstration-augmented programs from metric-accepted traces; compare its candidate against the baseline.
• Compile offline, serve without an optimizer loop: Optimization happens before deployment. Re-run it when your model, metric, or data distribution changes.
• Program structure can transfer; compiled state might not: When you change models, recompile and compare against release gates instead of blindly reusing old prompt artifacts.