Why do worked examples beat case studies in regulated industries?

A case study asserts that someone else succeeded. A worked example lets the evaluator reproduce the success themselves, with data in, inference out, and the audit trail intact. For a team bound by compliance rules, reproducibility is the class of evidence that survives internal review, because a claim the team can re-run moves the risk assessment from your marketing to their own eyes.

Separate the two classes of evidence

A case study is a story. It says a named or unnamed customer deployed a model, saw a result, and was happy. The reader has to trust the teller, and in a regulated industry the reader is not allowed to. The transcripts behind Theme 3 show the same scene repeatedly. A compliance reviewer reads a vendor case study and asks one question, which is whether the team can verify any of it. If the answer is no, the case study counts as marketing, and marketing does not go into a risk file.

A worked example is a procedure. It has four parts, and every part is checkable.

• The input data, either included or generated by a script the reader can run.
• The exact commands, in order, with versions pinned.
• The expected output, stated concretely enough that a reader knows whether they got it.
• The audit artifact each step leaves behind, e.g., a log file or a checksum.

The difference is who does the believing. With a case study, you believe the vendor. With a worked example, you believe your own terminal. A regulated team can put the second kind of evidence in front of an internal reviewer, because the reviewer can run it too. The claim stops being about your customer and starts being about their machine.

The transcripts add one more point. A reviewer who catches a vendor making one unverifiable claim starts to discount the verifiable ones too. So a single worked example does more for trust than a page of case studies, because it never asks for trust in the first place.

Work through what a reproduction costs

A worked example only beats a case study if the evaluator can actually run it, so the reproduction cost has to be low. Here is the arithmetic for a small open model, using Bonsai 8B in GGUF format as the example.

• Weight memory at 4-bit quantization: 8 billion parameters x 0.5 bytes = 4 GB.
• Working memory for the cache and overhead: add roughly 25 percent, so plan for about 5 GB.
• That fits in the RAM of an ordinary 16 GB laptop, with no GPU required. That estimate is derived from the parameter count, not measured, so state it the same way in your own material.

Now compare the evaluator's cost for each evidence class. Reading a case study takes ten minutes and produces nothing they can file. Running the worked example takes about the same ten minutes of active work, a model download of a few gigabytes, and it produces a transcript, an output file, and checksums they can attach to their own review. The active time is roughly equal, and only one of the two leaves an artifact.

The 4 GB figure is why the model choice is part of the argument. A worked example that needs an 80 GB GPU is a case study again for most evaluators, because they cannot run it. Pick the smallest model that shows the behavior you are claiming.

Convert a case study claim into a worked example

Take a claim in the form "Customer X ran our model on their documents and got usable summaries." Convert it column by column.

1. Replace the customer's data with stand-in data the reader can generate. Ship a script that writes ten synthetic documents of the same shape. The audit artifact is the script itself plus the checksum of its output.
2. Replace "ran our model" with the exact commands, including the model file, its revision, and the runtime version. The audit artifact is the shell transcript.
3. Replace "got usable summaries" with a concrete expected output, e.g., "each summary is under 100 words and names the document's date." The audit artifact is the output file.
4. Add a final step that hashes the inputs and outputs into one manifest, so the reviewer can prove later which files the run used.

The customer's identity, the one part of a case study a vendor most wants to show, is the one part the conversion drops. That is the point. The evaluator was never going to be allowed to rely on it.

Try it

This takes about 25 minutes, plus a model download of about 5 GB. No GPU is needed. The commands run on CPU, and they run faster on Apple Silicon or a CUDA GPU if you have one.

1. Pick one case-study-style claim you have published or read. Write it down in one sentence.

2. Download the model and pin the exact file:

   pip install -U "huggingface_hub[cli]"
   hf download prism-ml/Bonsai-8B-gguf --local-dir ./bonsai
   shasum -a 256 ./bonsai/*.gguf > manifest.txt
   

3. Run one inference and keep the transcript. This mirrors the first inference guide:

   llama-cli -m ./bonsai/<model-file>.gguf \
     -p "Summarize: <one paragraph of stand-in data>" \
     -n 128 2>&1 | tee run-01.log
   

4. Write the outline for your claim with the four parts: input data, exact commands, expected output, and the audit artifact each step leaves. Steps 2 and 3 above are your first two rows.

5. Hash the outputs into the manifest so the run is provable:

   shasum -a 256 run-01.log >> manifest.txt
   

Check yourself

• A skeptical reader gets your outline and nothing else. Can they execute it without contacting anyone? Expected answer: yes, because every step has a command and an expected artifact. If any step needs your customer's data or a call to your team, that step is still a case study.
• What does the manifest file prove, and what does it not prove? Expected answer: it proves which exact files went in and came out of the run, so nobody swapped them later. It does not prove the output is good. The expected output stated in the outline covers that.
• Why does the worked example drop the customer's name? Expected answer: the reviewer cannot rely on another company's identity in a risk file, so the name adds nothing the format needs. The reviewer's own run replaces it.

Next steps

• Read the regulated deployment case study and mark which of its claims you could convert with the four steps above.
• Work through the first inference guide so the commands in your worked example match the documented path.
• For the order in which a regulated buyer will read your worked example, see what a hospital CTO asks before running a model on-prem.

When you can do this, you can transform an assertion-based proof into a reproducible worked example with audit artifacts.