Skip to content
Tanner Goins

· 7 min read

Prompt Engineering Is Software Engineering

Prompts are production artifacts. They deserve version control, testing, review processes, and deployment strategies, just like any other code you ship.

By Tanner Goins

Laptop screen in a dark room showing JavaScript code in an editor with colorful syntax highlighting

Here’s something I’ve watched happen on three different teams now:

A developer writes a prompt. It works. They paste it into the codebase as a string literal. It ships. Weeks later, someone tweaks it to fix an edge case. Then someone else tweaks it to fix a different edge case. Then the original behavior breaks and nobody knows why.

Sound familiar?

We’ve spent decades building tools and practices for managing code changes: version control, code review, testing, CI/CD. But when it comes to prompts, most teams treat them like configuration strings rather than the production-critical logic they actually are.

A prompt change is a behavior change. It should be treated with the same rigor as any other code change. Here’s what that looks like in practice.

Prompts Are Code. Treat Them Like It.

A prompt defines the behavior of an LLM-powered feature. Change the prompt, and you change what the system does. That’s the definition of code.

But prompts have properties that make them trickier than traditional code:

  • Non-deterministic outputs. The same prompt can produce different results on different runs. This makes testing harder but not impossible.
  • Subtle behavioral shifts. A small wording change can cause a large behavior change, or no change at all. You can’t always predict which. This is exactly how AI products fail quietly.
  • Context sensitivity. A prompt that works perfectly with GPT-4o might fail with Claude, or even with a different version of the same model.
  • No compiler. There’s no syntax check that tells you a prompt is “wrong.” It will always produce something. The question is whether that something is correct.

These properties don’t mean prompts can’t be engineered rigorously. They mean the engineering practices need to be adapted.

Version Control Your Prompts

This is the lowest-hanging fruit, and most teams still don’t do it.

Don’t do this:

const response = await llm.complete({
  system:
    "You are a helpful assistant that summarizes documents. Keep summaries under 200 words. Focus on key findings and actionable insights. Use bullet points for clarity.",
  user: documentContent,
});

Do this instead:

prompts/
  summarizer/
    v1.0.0.txt
    v1.1.0.txt    # Added bullet point formatting
    v1.2.0.txt    # Refined for technical documents
    current.txt   # Symlink or pointer to active version

Or, if you prefer keeping prompts in code, at minimum extract them into named constants with version comments:

// prompts/summarizer.ts
export const SUMMARIZER_SYSTEM_PROMPT = `
You are a document summarization assistant.

## Instructions
- Produce summaries under 200 words
- Focus on key findings and actionable insights
- Use bullet points for clarity
- For technical documents, preserve specific metrics and data points

## Output Format
Return a JSON object with:
- "summary": string (the bullet-point summary)
- "confidence": number (0-1, your confidence in the summary quality)
- "key_terms": string[] (important terms from the document)
`;

The point isn’t the specific structure. It’s that prompt changes are tracked, diffable, and reversible, just like any other code change.

Review Prompts Like Pull Requests

When someone on your team changes a prompt, it should go through code review. But prompt review requires a different lens than code review.

What to look for in a prompt review:

  1. Intent clarity. Can you tell what the prompt is trying to accomplish? If the reviewer has to guess, the LLM will too.

  2. Edge case coverage. Does the prompt handle the inputs that actually appear in production, not just the happy path examples?

  3. Output format stability. If downstream code parses the LLM’s output, does the prompt constrain the format tightly enough? A prompt that says “return JSON” is less reliable than one that specifies the exact schema.

  4. Regression risk. Does fixing one edge case risk breaking existing behavior? This is where evals (covered in my previous post) become essential.

  5. Model assumptions. Does the prompt rely on capabilities specific to one model or version? If so, document that dependency.

A simple convention: require every prompt change PR to include at least three example input/output pairs showing the before and after behavior. This forces the author to demonstrate the change works and gives reviewers concrete evidence.

Test Your Prompts

Yes, you can test non-deterministic systems. You just need to adjust your expectations about what “passing” means.

Assertion-Based Tests (For Structured Outputs)

If your prompt produces structured output (JSON, classifications, extractions), you can write conventional tests:

describe("document summarizer", () => {
  it("produces valid JSON output", async () => {
    const result = await summarize(sampleDocument);
    const parsed = JSON.parse(result);
    expect(parsed).toHaveProperty("summary");
    expect(parsed).toHaveProperty("confidence");
    expect(parsed.summary.split(" ").length).toBeLessThan(200);
  });

  it("extracts key terms from technical documents", async () => {
    const result = await summarize(technicalDocument);
    const parsed = JSON.parse(result);
    expect(parsed.key_terms).toContain("latency");
    expect(parsed.key_terms).toContain("throughput");
  });
});

These tests won’t catch every quality issue, but they catch structural regressions: the prompt stopped returning valid JSON, or it’s ignoring the word limit.

Golden Dataset Tests (For Quality)

Maintain a set of representative inputs with expected outputs. Run the prompt against them periodically and score the results. I covered this in detail in my eval pipeline post, but the short version:

  • 20–50 examples is enough to start
  • Use LLM-as-judge for scoring when manual review doesn’t scale
  • Track scores over time to detect drift

Smoke Tests in CI

At minimum, run your prompts against 3–5 inputs in CI to verify they don’t crash, return valid formats, and produce reasonable-length outputs. This catches the catastrophic failures that slip through when someone edits a prompt and doesn’t test it locally.

Deploy Prompts Deliberately

A prompt change should not go straight to 100% of traffic. Treat prompt deployments like feature deployments:

Canary releases. Route 5% of traffic to the new prompt. Monitor quality metrics for 24 hours. If scores hold, ramp to 25%, then 50%, then 100%.

A/B testing. When you’re not sure if a prompt change is an improvement, run both versions simultaneously and compare. This is especially valuable for subjective quality dimensions where automated scoring is noisy.

Rollback plans. If you version your prompts, rolling back is trivial. Point the system at the previous version. If your prompts are embedded in code, rollback means a code revert. Design for the scenario where you need to undo a prompt change at 2 AM.

Environment parity. The prompt you test in staging should be identical to what runs in production. This sounds obvious, but I’ve seen teams with “staging prompts” that differ from production prompts in ways nobody remembers.

The Organizational Challenge

The hardest part of treating prompts as software engineering isn’t the tooling. It’s the culture shift.

Prompt writing feels informal. It feels like writing documentation or comments, something that’s “just words” rather than “real code.” That perception leads to casual changes, skipped reviews, and untested deployments.

Breaking this pattern requires two things:

  1. Make prompts visible. Put them in files that show up in diffs. Require PRs for changes. Add them to your team’s definition of “code that needs review.”

  2. Make quality measurable. If you can show that a prompt change caused a 5% drop in output quality, people start taking prompt changes seriously. If you can’t measure quality, every prompt change is a guess.

Start Here

If you’re managing prompts casually today, here’s a practical migration path:

Week 1: Extract all prompts from inline strings into dedicated files or constants. One prompt, one file.

Week 2: Set up a golden dataset for your highest-traffic prompt. Just 20 examples. Score the current prompt to establish a baseline.

Week 3: Require prompt changes to go through code review. Include example inputs/outputs in the PR description.

Week 4: Add smoke tests to CI. Verify prompts produce valid output formats on a handful of test cases.

That’s four weeks of incremental work, and you’ll have more rigor around your prompts than most teams shipping LLM features today.

Prompts aren’t magic incantations. They’re instructions to a system, and those instructions determine what your product does and how it behaves. They deserve the same engineering discipline as the code they’re embedded in.

Version them. Review them. Test them. Deploy them deliberately. Your future self, the one debugging a subtle regression at midnight, will thank you.

Keep reading