A/B testing prompts: how to test LLM prompts properly

Why guessing fails

Prompts are sensitive to wording in ways that are not obvious. Reordering instructions, adding one constraint or swapping an example can change output quality noticeably. And the same prompt gives a slightly different answer each time you run it, so one good result proves very little. Tweaking by feel makes things worse about as often as it makes them better.

How to A/B test LLM prompts, step by step

1. 1Write down what good looks like before you start: accuracy, tone, format, length. Decide it first, or you will judge whichever answer you happen to like.
2. 2Collect a test set of real inputs, around ten to twenty, including a few awkward ones: missing details, long inputs, unusual requests.
3. 3Write variant B so it differs from A in one thing only. Change five things and you learn nothing about which one mattered.
4. 4Run both variants on every input, with the same model and settings, more than once each, because the output varies from run to run.
5. 5Score the outputs blind: hide which variant wrote which, then pick the better of each pair against your standard.
6. 6Count the wins. Keep the winner as your new A, then change one more thing and test again.

What to test first

• An example of a good output. Adding one real example is often the biggest single improvement.
• Who it is for. Naming the reader ("a busy café owner") changes vocabulary and length.
• The output format: headings, bullets, a table, a word limit.
• The order of instructions. Put the most important rule first or last, not buried in the middle.
• Constraints: what to avoid, what to do when information is missing.

How many runs do you need?

There is no magic number, and anyone who quotes one is guessing. The honest rule: the smaller the difference between A and B, the more inputs you need to see it. If B does not win clearly across ten to twenty varied inputs, treat it as a tie and keep the simpler prompt. A big, obvious win shows up fast; a subtle one needs a bigger test set.

Scoring without fooling yourself

• Compare pairs, not scores out of ten. "Which of these two is better?" is far more consistent than rating each on its own.
• Hide the labels. Knowing which variant you wrote last biases you toward it.
• Use a second model as a judge if you have many outputs, but check a sample of its verdicts yourself. Models have their own preferences, such as favouring longer answers.
• Keep the losing outputs. They show you what to fix next.

Test on the model you will actually use

A prompt that wins on one model can lose on another, because models are trained differently and read instructions differently. Test on the model your tool or workflow really runs on. If you switch models later, run the test again.

Doing it on BizWax.Ai

PromptLine turns a rough prompt into a complete one in five stages, each done by a different model, and shows exactly what each stage changed. That makes it a quick way to produce a strong variant B, with every change visible so you know what you are testing. BizChat lets you run the same prompt against different models in one conversation, and AI Consensus shows where several models agree and where they split. Everything you run is saved to your Library, so last week's winning prompt is still there when you test the next change.