Prompt Optimization with DSPy

The last lesson left deploy-answerer-v1.2-fix-b eligible for canary review after it passed a frozen suite. At 09:17, a canary request like req_205 changes the wording of the rollback question; the hand-edited template flips from ABSTAIN to SERVE without a citation while latency and HTTP errors stay green. That is a prompt regression, not a useful signal to an on-call engineer. The next question is whether you can search instructions and few-shot demos against those same frozen cases, instead of rewriting the string until it feels right.

DSPy^{[1]Reference 1DSPy: Compiling Declarative Language Model Calls into Self-Improving Pipelines.https://arxiv.org/abs/2310.03714} turns that release problem into a search you can inspect. You declare the task interface, write a metric, and let an optimizer propose instructions and demonstrations for one configured language model. A separate holdout test still decides whether the compiled candidate is worth shipping. Compilation is empirical search: it can overfit its data, and it doesn't prove the program is correct.

Manual templates don't search

Hand-tuned deploy-answerer prompts leave four questions unanswered:

1. Model coupling. Wording that works on one hosted model can collapse on the next revision of the same alias.
2. Fragility. A small instruction change can flip a serve/abstain decision on req_205.
3. No search loop. Experiment tracking can compare two templates you already wrote. It doesn't propose the next instruction for you.
4. Hidden state. Last Tuesday's release mixed a prompt string, a model alias, a metric, and a dataset. Which combination shipped?

A DSPy program keeps control flow relatively stable and searches prompt state for one LM and one metric. That separation gives you a candidate to inspect, not a release decision. You still own evaluation and promotion.

A program makes three contracts explicit:

• What you want the LM to do (signatures)
• How to compose LM calls (modules)
• What good looks like (metrics)

With those contracts in place, an optimizer can propose instructions and few-shot examples. You then compare that candidate with the uncompiled baseline on held-out tickets.

Signatures, modules, and metrics

A signature is the task interface, not the prompt text. It names inputs, outputs, and field meanings the way a Python function signature names arguments. Ask what should stay stable when req_205 changes wording: evidence and question still enter, and an answer still leaves. DSPy fills in formatting, while later compilation can rewrite instructions and demonstrations around that boundary.

For deploy-answerer, the job is: given admitted ReleaseOps evidence and an engineer's question, either serve a cited claim or abstain. Use that boundary while reading the API snippets below. They need the dspy package, and compilation needs a configured LM. Tests after them stay on the standard library so you can check decisions without calling a model.

A module wraps a signature and holds compiled state such as demonstrations. dspy.ChainOfThought(GenerateAnswer) prepends a reasoning output field, calls the LM, and fills answer.^{[1]Reference 1DSPy: Compiling Declarative Language Model Calls into Self-Improving Pipelines.https://arxiv.org/abs/2310.03714} That rationale is model output, not the execution trace. A trace records predictor calls, inputs, and outputs while the program runs, so don't mistake a readable rationale for an audit log.

If you later swap the LM or edit a field description, keep the module structure, recompile, and evaluate. A previous artifact can be a useful baseline, but don't assume it transfers.

Why is a DSPy signature more than a prompt template?

Answer

A signature defines the task interface: inputs, outputs, and field meanings. DSPy can then compile prompts and demonstrations for that interface, while the program logic stays stable.

Ask what a green check should mean for req_205: a correct route isn't enough if SERVE lacks a citation. A metric turns that product rule into a score. It takes a labeled example and a prediction, often returns 0.0 or 1.0, and may run hundreds of times, so keep it cheap and aligned with behavior you plan to ship.

Exact string match is fine for a short route label such as SERVE or ABSTAIN. It's the wrong tool for the full answer sentence. The metric below scores two things: did the model pick the right route, and did a SERVE answer reuse a fact from the admitted evidence?

A cited SERVE scores 1.0. An unsupported SERVE keeps the route half and loses the citation half. A wrong-route ABSTAIN scores 0.0. dspy_answer_metric adapts the product rule to DSPy: the optimizer passes a labeled example, a prediction, and an optional execution trace, not three standalone strings.

For open-ended wording, you can add an LLM-as-a-judge term after you calibrate it on held-out human labels. Judge scores can vary across runs, prefer verbose answers, and shift with candidate order.^{[2]Reference 2Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena.https://arxiv.org/abs/2306.05685} Lower temperature can cut sampling variance. It doesn't remove those biases. Keep the deterministic route and citation checks for the behaviors you refuse to regress.

Search on validation, promote on holdout

Now the optimizer (older DSPy papers say teleprompter) has enough information to search. It takes the program, labeled examples, and metric, then returns a compiled copy. .compile() deep-copies the student, so the original module stays uncompiled.^{[1]Reference 1DSPy: Compiling Declarative Language Model Calls into Self-Improving Pipelines.https://arxiv.org/abs/2310.03714} Current docs center the word optimizer; both names refer to the same search.

This is discrete search over prompt state, not gradient descent on model weights. The parameters $\phi$ are instruction strings and sets of demonstrations. Hosted LM calls don't expose a gradient through that text, so DSPy uses random search, bootstrap filtering, or a surrogate-guided trial loop depending on the optimizer. A higher validation score means the metric preferred that candidate on observed cases; it doesn't mean model weights moved.

Formally, we want the parameter set $\phi$ that maximizes the expected metric $M$ over a dataset $D$:

$$\phi^* = \arg\max_{\phi} \mathbb{E}_{x \in D} [M(x, \text{Program}_\phi(x))]$$

$D$ here is the validation split the optimizer is allowed to see. In practice, the expectation is the sample mean over that split. Promotion still uses a separate test split, so selection and release measure different things.

What does $\phi$ represent in DSPy optimization?

Answer

It represents prompt-program choices such as instructions and demonstrations. The optimizer searches over those choices to maximize a validation metric, not to update model weights with gradients.

Split train, validation, and test before the optimizer sees any rows. Train supplies traces for bootstrapping, validation picks a candidate, and test makes the promotion decision. MIPROv2 will carve a validation split out of trainset if you omit valset. That's convenient for an experiment, but reusing that pool as holdout leaks the selection signal.

Predict before reading the next output: the validation winner needn't be the release winner. With two frozen tickets, the optimizer might try a zero-shot instruction, a few bootstrapped demos, or a rewritten instruction plus those demos. It keeps the candidate with the highest validation mean, then holdout can still favor the uncompiled baseline.

compiled_a wins validation at 0.95 against the baseline's 0.70, then scores 0.65 on holdout against 0.80. The gap is useful evidence: selection found a validation fit, while promotion tested transfer to unseen tickets.

BootstrapFewShot: traces that pass become demos

If the metric accepts one evidence trace and rejects another, should the accepted trace become a future prompt example? LabeledFewShot attaches labeled train rows as demos and doesn't call the LM during .compile(). BootstrapFewShot goes one step further: it runs a teacher program (often a copy of the uncompiled student) on the training set, keeps traces the metric accepts, and writes those traces into the candidate's demonstrations.

1. Bootstrap. Run the teacher on training tickets. Extra rounds copy the LM at temperature=1.0 to gather diverse traces.
2. Filter. Call metric(example, prediction, trace). A truthy score, or a score at or above metric_threshold, accepts the trace.
3. Assign. Up to max_bootstrapped_demos slots (default 4) get those accepted traces. Remaining slots up to max_labeled_demos (default 16) fill with raw labeled rows.

That loop only proves a trace passed your metric once. Compare the compiled copy with the uncompiled baseline on holdout before you keep it; a polished demo can still encode a training-set accident.

The library call looks like this. Compilation itself needs dspy.configure(lm=...) and spends LM calls, so the lab below simulates only the filter.

BootstrapFewShotWithRandomSearch (alias BootstrapRS) repeats that bootstrap with different seeds, scores each demo set on validation, and keeps the best. Reach for it when one bootstrap pass is too noisy and you can afford more compile budget, then retain the same holdout gate.

MIPROv2: joint instruction and demo search

BootstrapFewShot doesn't rewrite the instruction. If good demos still lead to the wrong route, MIPROv2 jointly searches instruction text and few-shot sets.^{[3]Reference 3Optimizing Instructions and Demonstrations for Multi-Stage Language Model Programshttps://arxiv.org/abs/2406.11695} Official docs describe three stages:

1. Bootstrap demo sets, the same rejection-sampling idea as above, producing num_candidates demo bundles.
2. Propose instructions with a prompt_model. The proposer sees a data summary, the program structure, the bootstrapped demos, and a sampled tip such as "be concise".
3. Search combinations with Bayesian optimization. The current DSPy implementation drives that loop with Optuna and a multivariate TPE sampler, so compile hosts need dspy[optuna]. Trials can score minibatches, then re-check the leader on the full validation set.

Default auto="light" sets candidate count, trial count, and validation cap for you. Start there. Manual mode is auto=None plus num_candidates on the constructor and num_trials on compile(...). Mixing auto="light" with num_candidates raises, and omitting num_trials in manual mode raises too.

The explicit split is a release guardrail. Pass trainset, valset, and a held-out test set you never hand to compile. Current DSPy builds a validation split from trainset when valset is omitted; convenient for exploration, unsafe as a release set.

As a default, instruction search can transfer farther than a large demo set. Demo-heavy candidates can memorize the train tickets you bootstrapped from. If validation is tiny or skewed, prefer instruction-heavy settings and keep a real holdout. A MIPROv2 run is zero-shot only when both demo sources are disabled at compile time: max_bootstrapped_demos=0 and max_labeled_demos=0.^{[4]Reference 4MIPROv2 optimizer implementationhttps://github.com/stanfordnlp/dspy/blob/3f06959ebbb80fb0bda001e1f802d70c2d5be7f7/dspy/teleprompt/mipro_optimizer_v2.py} Setting only the bootstrapped limit to zero still permits labeled demonstrations.

Which optimizer to try first

DSPy doesn't pick an optimizer for you. The auto knobs on MIPROv2 and GEPA set budget inside one algorithm; they don't compare algorithms for you.

GEPA maintains a population of programs, reads natural-language feedback from the metric, and samples parents from a Pareto frontier of per-example winners.^{[5]Reference 5GEPA: Reflective Prompt Evolution Can Outperform Reinforcement Learninghttps://arxiv.org/abs/2507.19457} Across its six paper tasks, it reports beating GRPO by 6% on average (up to 20%) with up to 35× fewer rollouts, and beating MIPROv2 by over 10% (including +12% accuracy on AIME-2025). Those numbers select GEPA on that benchmark, not on deploy-answerer.

A GEPA metric can return a float, but the richer contract is dspy.Prediction(score=..., feedback="..."): the reflection LM can turn feedback into its next instruction proposal. A bare float still runs; the proposer then sees a generic "this trajectory got a score of n" caption. Compare measured holdout gain and compile cost with BootstrapFewShot or MIPROv2 before you adopt it.

Save JSON state, not a pickle from the internet

A compiled program is only useful if you can reload the prompt state later. DSPy has two save paths, and mixing them up makes an otherwise reproducible compile hard to deploy.

1. State-only JSON. program.save("compiled_a.json") writes predictor state: instructions, demonstrations, signature metadata, and sanitized reconstruction state for an attached LM. Current DSPy LM state includes its model identifier and safe configuration, but excludes API keys. Loading recreates that per-predictor LM, and a later global dspy.configure(lm=...) doesn't replace it.^{[6]Reference 6DSPy program and LM state serializationhttps://github.com/stanfordnlp/dspy/tree/3f06959ebbb80fb0bda001e1f802d70c2d5be7f7/dspy} Recreate DeployAnswerer() in code, call .load("compiled_a.json"), and use the loaded LM unless this release deliberately changes the model. For an intentional override, call program.set_lm(runtime_lm) and evaluate that artifact-model pair before promotion.
2. Whole-program directory. program.save("compiled_a/", save_program=True) writes program.pkl plus dependency metadata. Restore with dspy.load("compiled_a/", allow_pickle=True). That path deserializes cloudpickle, so treat the directory like executable code and only load artifacts you trust.

Prefer JSON for ordinary releases. It's readable, diffable, and doesn't execute code on load. It still isn't a secret store or a complete release manifest. Current DSPy removes API keys when dumping LM state and strips unsafe endpoint fields during ordinary load; restoring custom LM classes or unsafe endpoint state requires an explicit trusted-state opt-in.^{[6]Reference 6DSPy program and LM state serializationhttps://github.com/stanfordnlp/dspy/tree/3f06959ebbb80fb0bda001e1f802d70c2d5be7f7/dspy}

The last two lines capture the precedence trap. A loaded predictor carries its saved LM, so changing the global default doesn't move it. set_lm is the explicit release action that recursively replaces predictor LMs. Treat that call as a new tested pairing, not as harmless runtime wiring.

Compile offline, serve a frozen artifact

.compile() can spend a large LM budget. Run it in an offline job, then let request handling load the tested artifact and answer. If an optimizer loop reaches the request path, latency, cost, and even prompt state now depend on traffic.

Before trusting average holdout lift, inspect critical slices. If rollback guidance drops below its floor, block the candidate even when the mean improved.

What must be versioned after a DSPy compile run?

Answer

Version the compiled artifact, target model, dataset split, metric, optimizer settings, and source program. Without those pieces, you can't explain or roll back a production quality change.

A stable DeployAnswerer interface lets you run a new compile when the underlying model changes. It doesn't make old JSON model-independent. Instruction phrasing and demos can be specific to one snapshot. A provider-side update is a new experiment, even when the alias string didn't change. Keep a compile record keyed by program version and exact model revision.

That record is what the next lesson packages into a release bundle. The JSON file is one field; compile run id, model pin, eval suite hash, and rollback target complete the release evidence.

When compilation burns budget

When compile burns budget, diagnose the symptom before increasing the budget. Each failure below points to a different contract.

Exact match on long answers

Almost every prediction scoring 0 means bootstrap keeps nothing and MIPROv2 wanders. The usual cause is scoring open-ended text as if it were a class label: "ABSTAIN because no ReleaseOps record exists" and "ABSTAIN|no ReleaseOps record" can express the same decision. Score route and citation deterministically, then add a calibrated judge only for wording rules can't check.

Waiting for thousands of labels

If you postpone .compile() until the dataset looks like a fine-tune set, you're importing weight-training habits into prompt search. Start with a small, representative split, watch slice coverage, and add tickets where evaluation is unstable. Don't invent a required sample count before measuring the task.

Never reading the compiled prompt

If holdout looks fine but production fails and nobody can say which demos shipped, you treated .compile() as a black box. After compilation, inspect recent LM calls with dspy.inspect_history(n=...). Edge-case demos or misleading instructions point back to coverage or metric design, so fix those and rerun the comparison.

Hidden setup mistakes

Finally, check setup before blaming search. Calling modules without a metric and .compile() is prompt formatting, not optimization. MIPROv2 rejects num_candidates while auto is still "light"; manual mode needs auto=None and num_trials. Its compile path imports Optuna, so install dspy[optuna] on compile hosts.

A five-ticket validation set can memorize: tiny val plus a large demo count is how compiled_a beats the baseline in-sample and loses on holdout. JSON state doesn't recreate DeployAnswerer, so load it into a compatible class. It can recreate an attached predictor LM from sanitized state; use set_lm when a reviewed release intentionally replaces that model. dspy.load(..., allow_pickle=True) executes cloudpickle; only load trusted directories. Heavy optimizers belong in batch jobs, never inside a request handler.

Key takeaways

• DSPy searches instructions and demonstrations for a declared program. It doesn't replace held-out evaluation.
• Signatures name the interface. Modules own control flow. Optimizers fill prompt state. .compile() returns a new copy.
• BootstrapFewShot turns metric-accepted traces into demos. MIPROv2 also rewrites instructions. Compare every candidate with the baseline on a split the optimizer never saw.
• Save JSON state for ordinary releases. Recreate the Python class, then load its predictor and sanitized LM state. Keep credentials outside the artifact and use set_lm only for an evaluated model override.
• Version the artifact with the model revision, metric, split hashes, optimizer settings, and source program. That's the object the next lesson will pin inside a release bundle.