# PrismML Developer Documentation - full content. Index: https://yourwildcard.ai/llms.txt --- --- title: PrismML Developer canonical_url: https://yourwildcard.ai/ --- # Home PrismML is a family of small open-weight language models, led by Ternary Bonsai 27B. Work through these docs and you will be able to run the 27B checkpoint on hardware you already own, inspect measured local traces, and explain where its compressed weights help without overclaiming. - [Start with Bonsai 27B](https://yourwildcard.ai/docs/prismml/bonsai-27b.md): Run the public ternary checkpoint, read the local M4 Pro traces, and choose an MLX workflow. - [Read what people are saying](/blog/bonsai-27b-ecosystem-reactions): The Information, PrismML, Khosla team amplification, launch posts, and the claims that still need testing. - [Understand the ecosystem](https://yourwildcard.ai/docs/start-here/orientation.md): See how data, models, compression, runtimes, and devices fit together, and where PrismML sits. - [Contribute your first page](https://yourwildcard.ai/docs/contribute/contributor-guide.md): Found a gap or an error? Here is how to fix it. ## Ask instead of searching The fastest way through these docs is to ask them. [Ask the docs](/chat) runs 1-bit Bonsai 1.7B in your own browser over WebGPU: type a question, watch the inference trace show how the answer was made, and follow the source links it cites into the site. Nothing you type leaves your device. It is also the search bar: press the search box anywhere and hand your query to Bonsai from there. ## The main story: Ternary Bonsai 27B The public MLX checkpoint puts a 27B parameter model in an 8.49 GB package. In our controlled LM Studio runs on a 24 GB M4 Pro Mac, it decoded at 21.55 and 22.67 tokens per second at a 4,096 token context. It also returned a valid structured tool call. A reasoning check showed the important limit: a 16 token budget produced reasoning but no answer, while a 256 token budget returned `42` after 191 reasoning tokens. Start with the [Bonsai 27B field guide](https://yourwildcard.ai/docs/prismml/bonsai-27b.md), then read the [26 focused lessons](/blog) and the [link-first ecosystem roundup](/blog/bonsai-27b-ecosystem-reactions). The roundup centers [The Information's report](https://www.theinformation.com/articles/khosla-backed-startup-claims-breakthrough-largest-ever-ai-model-iphone), then separates PrismML's claims, investor amplification, platform reactions, and our local evidence. ## Choose how you connect - **With a coding agent.** Point Claude Code, Cursor, or any agent at [/llms.txt](https://yourwildcard.ai/llms.txt); every page is raw Markdown if you append `.md` to its URL. [Use these docs with AI](https://yourwildcard.ai/docs/start-here/use-with-ai.md) has a paste-ready starting prompt. - **By hand.** Start with the [llama.cpp recipe](https://yourwildcard.ai/docs/build-and-run/bonsai-llamacpp.md) and you will have a local model answering prompts in minutes. ## What you will be able to do - **Run** Ternary Bonsai 27B locally with MLX or LM Studio and serve it as an OpenAI-compatible API. - **Measure** tokens per second, time to first token, memory, and energy on your own device, in a format others can compare against. - **Verify** the whitepaper's claims, from the 1.15 GB file size to the benchmark scores, against official sources and your own runs. - **Explain** the results to your community with the baselines and caveats stated, using the same numbers the paper reports. ## Check what you need Pick the environment that matches how deep you want to go. Every path helps benchmark Bonsai. | Environment | Good for | Prerequisites | | --------------------------- | ------------------------------------------------------------- | ----------------------------------------------------------------------------------------- | | Local machine | Running Bonsai 27B, measuring speed and memory, offline demos | Apple silicon Mac with 24 GB unified memory recommended, plus at least 12 GB of free disk | | Notebook (Colab or Jupyter) | Re-running single benchmarks such as GSM8K | A GPU runtime and [EvalScope](https://github.com/modelscope/evalscope) | | Cloud GPUs | Reproducing the paper's full evaluation matrix | The paper's setup: EvalScope v1.4.2, vLLM 0.15.1, H100-class GPUs | The [whitepaper benchmarks page](https://yourwildcard.ai/docs/prismml/whitepaper-benchmarks.md) explains what to run in each environment and what the paper's numbers should look like when you get there. ## Understand the whitepaper benchmarks The goal of these docs is an ecosystem that understands the paper's results, not just repeats its headlines. The [1-bit Bonsai 8B whitepaper](https://yourwildcard.ai/whitepaper/1-bit-bonsai-8b-whitepaper.pdf) reports a 1.15 GB model scoring 70.5 across six benchmarks, generating up to 8.4x faster than FP16, and using 4x to 6x less energy per token. The [benchmarks guide](https://yourwildcard.ai/docs/prismml/whitepaper-benchmarks.md) walks through what each of the six tests measures, where Bonsai wins, where it gives up points, and how to check the numbers yourself. You can trace any claim to its source in the [claim and source matrix](https://yourwildcard.ai/docs/prismml/claim-source-matrix.md). ## Learn how it works underneath Two deployed courses go deeper than the guides on this site. Take them in either order. - [Inference The Hard Way](https://chaiwithjai.github.io/inference-the-hard-way/) teaches how Bonsai inference works, from the tokenizer and KV cache through kernels, bandwidth, and energy. - [Kubernetes The Hard Way](https://bootstrap-your-own-k8s.netlify.app) teaches the infrastructure under production serving: machines, certificates, etcd, the control plane, workers, and networking. The [technical guides](https://yourwildcard.ai/docs/technical-guides/kv-cache.md) cover the same concepts in shorter form, starting with the KV cache. ## Next steps - [Run and measure Ternary Bonsai 27B](https://yourwildcard.ai/docs/prismml/bonsai-27b.md) - [Read the Bonsai 27B ecosystem roundup](/blog/bonsai-27b-ecosystem-reactions) See something wrong? [Fix it](https://yourwildcard.ai/docs/contribute/contributor-guide.md). --- --- title: Orientation status: published audience: DevRel owner: DevRel maintainer source_tier: mixed benchmark_status: not_applicable last_reviewed: 2026-07-07 canonical_url: https://yourwildcard.ai/docs/start-here/orientation --- # Orientation PrismML is a family of small open-weight language models built to run on ordinary hardware such as laptops, phones, and efficient servers. On this page you will learn what the supply chain around those models is, and you will find the fastest route to running one yourself. ## Understand the supply chain A model file on its own does nothing. Before you can type a prompt and get an answer, several tools have to do their part. Someone trains the model, someone compresses the weights so they fit on a small device, a runtime loads them, kernels do the math on your specific chip, and an evaluation loop tells you whether the output is any good. This wiki calls that chain of tools the supply chain, and each stage gets its own set of pages here. This is the short version, and the [supply-chain layers page](https://yourwildcard.ai/docs/ecosystem/supply-chain-layers.md) breaks the same chain into nine layers. For example, when you run the Bonsai model on a laptop with llama.cpp, you touch four stages of the chain at once: the compressed weights on disk, the llama.cpp runtime that loads them, the CPU or GPU kernels that execute them, and the tokens per second you measure at the end. The diagram below shows every stage and how they connect. ## Why the wiki centers on PrismML PrismML positions its models around concentrated intelligence, which means packing as much capability as possible into weights small enough for local devices. That positioning makes the tradeoffs of the whole supply chain concrete. Model quality still counts, and these pages also treat memory use, energy use, runtime compatibility, and device-local deployment as primary subjects rather than footnotes. Two other viewpoints shape what the wiki covers: - **Ion Stoica.** His work centers on open-source AI systems, distributed compute, serving, and evaluation infrastructure. Because of that, these pages document shared primitives, reproducible systems paths, and benchmark methodology. - **Vinod Khosla.** He argues that intelligence should become cheaper and more abundant, and less constrained by energy or datacenter bottlenecks. Because of that, these pages track intelligence per dollar, per watt, per gigabyte, and per device. ## Know what to trust here Every canonical page cites durable evidence such as official docs, repos, papers, model cards, release posts, benchmark logs, or reproducible device reports. Claims that only come from social posts stay labeled as unconfirmed until someone verifies them against a durable source. When a page shows commands we have not yet run on hardware, the page says so in plain words. ## Next steps - [Run Bonsai on your machine with llama.cpp](https://yourwildcard.ai/docs/build-and-run/bonsai-llamacpp.md). This is the fastest way to see the whole supply chain work end to end. - [Read the supply-chain layers guide](https://yourwildcard.ai/docs/ecosystem/supply-chain-layers.md) if you want the concepts before the commands. See something wrong? [Fix it](https://yourwildcard.ai/docs/contribute/contributor-guide.md). --- --- title: Audience And Standards status: published audience: DevRel owner: Docs librarian source_tier: internal_standard benchmark_status: not_applicable last_reviewed: 2026-07-07 canonical_url: https://yourwildcard.ai/docs/start-here/audience-and-standards --- # Audience and standards This wiki explains the tools that take a small open-weight language model from downloaded weights to a running product, and it is written for people who build and explain AI products. Read this page to see whether the wiki fits your work and what you can trust when you read it. ## Find your starting point You will get the most from these docs if one of these describes you. - **You explain AI products.** You can talk about models with customers, but you may not know inference internals. The technical guides teach you how a model actually runs so your claims stay accurate. - **You build demos and sample apps.** The build-and-run recipes give you commands to run a model on your own machine and a way to record what you measured. - **You maintain documentation or benchmarks.** The device report schema and the sources pages show how we record evidence so someone else can reproduce it. - **You run an ecosystem project.** The ecosystem entries show where runtimes and tools fit in the chain from weights to product, so you can see where yours fits too. If none of these fit, start with the supply-chain layers page. It defines the whole chain in one place. ## Trust what you read Every page here follows the same rules, so you can rely on the same things everywhere. - Every command comes from an upstream source we link to. When we have not yet run a command on hardware, the page says so in plain words. - Every benchmark number states the task, model, runtime, device, and date. A number without that scope does not appear. - Social media posts are never treated as evidence. When a claim comes from an unconfirmed post, the page labels it as unconfirmed. - Open questions are stated as open questions, never as facts. - Product names and roadmap claims appear only when the vendor has published them. For example, the Bonsai recipe lists build commands that come from the upstream llama.cpp fork, and it says plainly that no one has yet recorded a run on hardware. ## See the standard we follow We model the structure of these docs on four documentation sets: the JobsByCulture DevRel career guide, I'd Rather Be Writing, MDN, and Rails Guides. From them we borrow the habits that help a reader most. - Start from the reader's work context and make the practical output explicit. - Move from concept, to workflow, to measurement, to maintenance. - Keep page types separate, e.g., an overview page does not double as a reference page. - Get you productive quickly, then offer deeper paths. ## Next steps - [Run Bonsai on your machine](https://yourwildcard.ai/docs/build-and-run/bonsai-llamacpp.md) - [Supply-chain layers](https://yourwildcard.ai/docs/ecosystem/supply-chain-layers.md) See something wrong? [Fix it](https://yourwildcard.ai/docs/contribute/contributor-guide.md). --- --- title: Wiki Contract status: published audience: Maintainers owner: Documentation PM source_tier: internal_standard benchmark_status: not_applicable last_reviewed: 2026-07-07 canonical_url: https://yourwildcard.ai/docs/start-here/wiki-contract --- # Wiki contract The wiki contract is the checklist a maintainer uses to decide whether a page in this wiki is finished. Use it to accept or reject a delivered page based on evidence, because a site that builds and deploys is not the same as a site whose pages are correct. ## Check a page against the contract A page passes the contract when you can point to evidence for each check below. For example, the Bonsai llama.cpp recipe passes the claim check because every command in it links to the upstream llama.cpp documentation, and it passes the runtime check because a recorded run on an M4 Pro is attached to the page. Most other recipes still carry "not verified" until someone attaches theirs. | Check | Evidence you need to see | | --- | --- | | Site structure | The page lives at a nested `/docs/...` route with sidebar navigation, search, a table of contents, callouts, and code fences. | | Source coverage | Every major source is mapped to a route or explicitly excluded, per the matrix below. | | Claim hygiene | Every PrismML claim appears in the dated [claim and source matrix](https://yourwildcard.ai/docs/prismml/claim-source-matrix.md) before it appears in prose. | | Runtime proof | A recipe either includes device and runtime logs from a run, or says plainly that no one has run its commands yet. | A page must also do a job for a reader. Someone should be able to use it to explain a concept, run a recipe, verify a claim, or teach the material. If you cannot name the job, reject the page. Deployment only proves the site can be served. Accept the wiki only after each page passes this contract. ## Map sources to pages Each source document must land on a public page or be excluded on purpose, so nothing disappears silently. Here is the current map. | Source | Public page | State | | --- | --- | --- | | `README.md`, `wiki-index.md`, `wiki-information-architecture.md` | Home, Orientation, Supply-chain layers | Moved over in full. | | `ecosystem-map/prismml.md` | PrismML model family, claim and source matrix | Partly moved. Some claims still need source review. | | `build-and-run/prismml-bonsai-llamacpp.md` | Bonsai with llama.cpp | Moved over. The commands come from the upstream docs, and we have not yet run them on hardware. | | `benchmarks/device-report-template.md` | Evaluation harness | Moved over in full. | | `governance/source-policy.md` | Source policy | Moved over in full. | | `contribution-backlog.md` | Contribution backlog | Moved over in full. | | `x-comet-research-notes.md` | Claim and source matrix discovery queue | Partly moved. The X posts stay labeled as unconfirmed. | | Inference The Hard Way labs 03 to 07 | KV cache, prefill and decode, weights, kernels, bandwidth | Moved over in full. | | `in-house-ml.md` | None | Excluded until the source owner approves publication. | ## Fill in the page contract When you add or review a page, fill in this template and keep it with the page's issue. It records who the page is for, where its facts come from, and how someone checks them. ```text route: audience job: primary question: source inputs: required visual or table: verification check: known gaps: owner: review cadence: ``` ## Next steps - Read the [source policy](https://yourwildcard.ai/docs/sources/source-policy.md) for the rules on which sources a claim may cite. - Trace individual claims in the [claim and source matrix](https://yourwildcard.ai/docs/prismml/claim-source-matrix.md). - Follow the [contributor guide](https://yourwildcard.ai/docs/contribute/contributor-guide.md) to submit or review a page. --- --- title: Use These Docs With AI status: published audience: DevRel owner: DevRel maintainer source_tier: internal_canonical benchmark_status: not_applicable last_reviewed: 2026-07-07 canonical_url: https://yourwildcard.ai/docs/start-here/use-with-ai --- # Use these docs with AI These docs are built for coding agents as much as for people. Point Claude Code, Cursor, or any agent at one URL and it can read every page as clean Markdown. ## Give your agent the index The whole site is indexed for machines at two URLs: - [`/llms.txt`](https://yourwildcard.ai/llms.txt) lists every page with a one-line description, grouped by section. Agents read this first to decide what to fetch. - [`/llms-full.txt`](https://yourwildcard.ai/llms-full.txt) is the full text of every page in one file, small enough to fit in a large context window. Every docs page also has a raw Markdown twin. Append `.md` to any page URL: ```text https://yourwildcard.ai/docs/prismml/whitepaper-benchmarks.md ``` ## Start a coding agent on Bonsai Paste this into Claude Code, Cursor, or your agent of choice as the first message: ```text Read https://yourwildcard.ai/llms.txt and use it as your map of the PrismML docs. Then get Bonsai 8B running on this machine by following https://yourwildcard.ai/docs/build-and-run/bonsai-llamacpp.md Record tokens per second, time to first token, and peak memory, and format the result using the device report schema page. ``` The agent gets the model file from the [Bonsai 8B GGUF model card](https://huggingface.co/prism-ml/Bonsai-8B-gguf) (1.16 GB, Apache 2.0). Before it downloads anything, have it check whether the file already exists on disk; weights are large and often already local. ## Trust the metadata Every raw Markdown page carries frontmatter your agent can parse before trusting the prose: ```yaml status: verified # verified | source_checked | draft | not_run source_tier: primary # how close the evidence is to an official source benchmark_status: partial # not_run | partial | reproducible | published last_reviewed: 2026-07-06 ``` Tell your agent the rule the site itself follows: numbers with `status: verified` trace to an official source; anything labeled `discovery` came from social posts and is not yet confirmed. The [claim and source matrix](https://yourwildcard.ai/docs/prismml/claim-source-matrix.md) is the machine-checkable ledger behind that rule. ## Know what the site does not have There is no MCP server and no search API; the site is static files. That is deliberate: static Markdown plus an index is the most portable agent interface, and every major docs platform we benchmarked (Vercel, LangChain, Hugging Face) converges on this shape. ## Next steps - [Run Bonsai with llama.cpp](https://yourwildcard.ai/docs/build-and-run/bonsai-llamacpp.md), the page your agent will follow. - [Whitepaper benchmarks](https://yourwildcard.ai/docs/prismml/whitepaper-benchmarks.md), the numbers your agent should compare its run against. See something wrong? [Fix it](https://yourwildcard.ai/docs/contribute/contributor-guide.md). --- --- title: Inference The Hard Way status: published audience: DevRel owner: Developer advocate source_tier: primary_course benchmark_status: partial last_reviewed: 2026-07-07 canonical_url: https://yourwildcard.ai/docs/learning-paths/inference-the-hard-way --- # Inference The Hard Way Inference The Hard Way is a lab course that shows how a language model goes from bytes on disk to tokens on screen. Work through it here to learn where inference spends its time and memory, and to check each claim yourself with bytes on disk, kernel code, cache growth, and benchmark numbers instead of marketing language. By the end of the course you can explain how a compressed model runs, and you can back the explanation with measurements from your own machine. [Start the course](https://chaiwithjai.github.io/inference-the-hard-way). This page tells you which order to take the labs in and which guide on this site pairs with each lab. ## Follow the labs in this order 1. Run Lab 00 and note your device details, e.g., chip, memory, and OS. You will want them later when you compare benchmark numbers. 2. Do the KV cache, prefill vs decode, and bandwidth ledger labs first, because they explain where the speed and memory costs come from. 3. Do the weights on disk and kernel reading labs next to see why "1-bit" is a storage format and an execution path, not only a smaller download. 4. Finish with the reproduction and capstone labs, which have you reproduce a whitepaper result and explain it to someone else. ## Pair each lab with its guide Most labs have a short guide on this site that covers the same concept. Read the guide before or after the lab, whichever helps you more. For example, Lab 03 has you measure how the KV cache grows with context length, and the [KV cache](https://yourwildcard.ai/docs/technical-guides/kv-cache.md) guide explains why that growth happens. | Course module | Guide on this site | | --- | --- | | Lab 03 KV cache | [KV cache](https://yourwildcard.ai/docs/technical-guides/kv-cache.md) | | Lab 04 prefill vs decode | [Prefill vs decode](https://yourwildcard.ai/docs/technical-guides/prefill-vs-decode.md) | | Lab 05 weights on disk | [Weights on disk](https://yourwildcard.ai/docs/technical-guides/weights-on-disk.md) | | Lab 06 reading the kernel | [Kernel reading](https://yourwildcard.ai/docs/technical-guides/kernel-reading.md) | | Lab 07 bandwidth ledger | [Bandwidth ledger](https://yourwildcard.ai/docs/technical-guides/bandwidth-ledger.md) | Labs 01 and 02, the tokenizer and the forward pass, do not have guides yet. ## Check your understanding Before you move on, you should be able to say this without notes: ```text Prefill and decode are different physical regimes. Prefill batches prompt tokens and is more compute-bound. Decode generates one token at a time and streams weights through memory, so compressed weights mostly help decode. The KV cache avoids recomputing past keys and values, but it becomes the next memory wall as context grows. ``` If any sentence in that summary feels shaky, redo the lab that covers it. ## Next steps - [Start the course](https://chaiwithjai.github.io/inference-the-hard-way) at Lab 00. - Read the [KV cache](https://yourwildcard.ai/docs/technical-guides/kv-cache.md) guide, the first technical guide on this track. - After the bandwidth ledger lab, check your predicted decode ceiling against the [Bandwidth ledger](https://yourwildcard.ai/docs/technical-guides/bandwidth-ledger.md) guide. See something wrong? [Fix it](https://yourwildcard.ai/docs/contribute/contributor-guide.md). --- --- title: Kubernetes The Hard Way status: published audience: DevRel owner: Developer advocate source_tier: primary_course benchmark_status: not_applicable last_reviewed: 2026-07-07 canonical_url: https://yourwildcard.ai/docs/learning-paths/kubernetes-the-hard-way --- # Kubernetes The Hard Way Kubernetes The Hard Way is a free course that walks you through building a working Kubernetes cluster by hand, one component at a time, from machines and certificates through etcd, the control plane, workers, networking, and smoke tests. Follow this path to learn what sits underneath every production model-serving deployment, so you can explain and debug the stack that runs vLLM, SGLang, or Ray Serve. Start the course here: [bootstrap-your-own-k8s.netlify.app](https://bootstrap-your-own-k8s.netlify.app). The source lives at [github.com/ChaiWithJai/kubernetes-the-hard-way](https://github.com/ChaiWithJai/kubernetes-the-hard-way). In production, builders deploy serving stacks like vLLM, SGLang, and Ray Serve on Kubernetes. If you cannot explain certificates, etcd, or pod networking, you cannot debug a builder's "my model server won't start" question past the first abstraction. This course teaches those components. ## Learn what each module makes visible The course builds the cluster in a fixed order, and each module exposes one layer that production serving depends on. Certificates come before the control plane because every component authenticates with TLS. etcd comes next because it holds all cluster state. Then you bootstrap the control plane that `kubectl` talks to, the worker nodes where model-server containers run and GPU device plugins attach, and the pod network that lets replicas of a model server reach each other and the load balancer. The course ends with smoke tests and cleanup, which teach the habit of proving each layer works before you trust the next one. | Module | What it makes visible | | --- | --- | | Prerequisites and machines | Node inventory, SSH access, and OS assumptions. | | Certificates and kubeconfigs | Why every component authenticates with TLS, and where the certificates come from. | | etcd | The store that holds all cluster state, and its failure modes. | | Control plane | What `kubectl` talks to, and how workloads get placed. | | Worker nodes | Where model-server containers run, and where GPU device plugins attach. | | Pod networking | How replicas of a model server reach each other and the load balancer. | | Smoke tests and cleanup | Proof that each layer works, and teardown you can repeat. | This topic list comes from the course's own materials. We have not yet transcribed the exact module titles from the deployed reader, so wording may differ slightly there. We also have not confirmed which Kubernetes version the course currently targets, or whether it covers GPU node setup or stops at CPU workers. ## Work the path in order 1. Work the prerequisites and machines modules. Record your environment the same way the [Bonsai recipe](https://yourwildcard.ai/docs/build-and-run/bonsai-llamacpp.md) environment record does. 2. Complete the certificate and etcd modules before the control plane, because they explain why the control plane's configuration looks the way it does. 3. Bootstrap the control plane, then workers, then networking, in that order. Do not skip ahead when a smoke test fails. Fixing the failure is how you learn what the layer does. 4. Run the final smoke test, then run cleanup. We have not yet recorded a full run of this course, so we cannot tell you the total time it takes or which versions you will end up with. The commands come from the course itself, which is actively deployed. ## Pair it with Inference The Hard Way The two Hard Way courses split the stack at the container boundary. [Inference The Hard Way](https://yourwildcard.ai/docs/learning-paths/inference-the-hard-way.md) explains how a model turns bytes on disk into tokens on screen, and why decode throughput is memory-bound. Kubernetes The Hard Way explains how that model server gets scheduled, networked, and kept alive in production. Take Inference The Hard Way first if your job is explaining models. Take Kubernetes The Hard Way first if your job is supporting the builders who deploy them. Both courses teach the same habit. Do not trust an abstraction until you can explain what it hides. ## Next steps - [Start the course](https://bootstrap-your-own-k8s.netlify.app) and build the cluster module by module. - [Run Bonsai with llama.cpp](https://yourwildcard.ai/docs/build-and-run/bonsai-llamacpp.md) to get a model server you could deploy on the cluster you just built. - [Take Inference The Hard Way](https://yourwildcard.ai/docs/learning-paths/inference-the-hard-way.md) to learn what happens inside the model server. See something wrong? [Fix it](https://yourwildcard.ai/docs/contribute/contributor-guide.md). --- --- title: Designing The Docs Chat status: verified audience: DevRel owner: DevRel maintainer source_tier: internal_canonical benchmark_status: partial last_reviewed: 2026-07-07 canonical_url: https://yourwildcard.ai/docs/learning-paths/designing-the-docs-chat --- # Designing the docs chat The docs chat agent is a worked example of the two Hard Way courses: Inference The Hard Way chose the model and the prompt shape, and Kubernetes The Hard Way ruled out every deployment except the browser. This page shows the arithmetic and the decisions so you can reuse them. ## Let deployment constraints pick the architecture This site deploys on Netlify as static files. That single fact eliminates most chat architectures, and [Kubernetes The Hard Way](https://bootstrap-your-own-k8s.netlify.app) is why we could see it quickly: after you have built a serving cluster by hand once, you know exactly what a server-side model needs and what it costs. | Option | What it needs | Verdict | | --- | --- | --- | | Netlify static hosting | Nothing. Files only. | No compute at all, so no server-side model | | Netlify Functions | CPU-only serverless, short timeouts, cold starts | An 8B model on a cold CPU function is seconds per token. Not viable | | A GPU server we run | Everything the course teaches: machines, TLS, ingress, autoscaling, plus a GPU bill that scales with users | Viable but it turns a free community wiki into an ops project | | WebGPU in the reader's browser | A 290 MB one-time download and a WebGPU browser | Zero infrastructure, zero marginal cost, private by construction | The browser wins on every axis that matters for a community docs site: each new reader brings their own GPU, so the system scales by definition, and questions never leave the device. ## Let inference arithmetic pick the model [Inference The Hard Way](https://chaiwithjai.github.io/inference-the-hard-way/) teaches you to budget a deployment in bytes before writing code. The budget for a browser is strict: the download, the memory, and the bandwidth all belong to someone else's laptop. - **Download.** 1-bit Bonsai 8B is 1.16 GB; 1-bit Bonsai 1.7B is 290 MB in the [ONNX build](https://huggingface.co/onnx-community/Bonsai-1.7B-ONNX). A 290 MB one-time download is acceptable for a docs page; 1.16 GB is a hard sell. - **KV cache.** Bonsai 1.7B has 28 layers, 8 KV heads, and head dimension 128, so the cache costs 2 x 28 x 8 x 128 x 2 bytes, about 112 KB per token. A 2,048-token conversation costs about 0.22 GB on top of the weights. The [KV cache guide](https://yourwildcard.ai/docs/technical-guides/kv-cache.md) walks this formula. - **Prefill.** Every question re-sends retrieved doc excerpts, and prompt processing is the price you pay before the first token appears. We cap excerpts near 2,400 characters per page, roughly 600 tokens, to keep time to first token tolerable on integrated GPUs. The [prefill vs decode guide](https://yourwildcard.ai/docs/technical-guides/prefill-vs-decode.md) explains why this phase dominates. - **Bandwidth.** Decode streams the weights once per token, so a 290 MB model on even a modest GPU has a comfortable ceiling. This is the [bandwidth ledger](https://yourwildcard.ai/docs/technical-guides/bandwidth-ledger.md) argument working in our favor: 1-bit weights are what make browser inference reasonable at all. ## Let an eval pick the prompt We wrote four gold questions with known answers in the docs, then ran the exact chat prompt through Bonsai 8B locally with llama.cpp before shipping anything. The eval caught two real bugs: 1. **Retrieval failure.** Raw keyword counting let the word "Bonsai" drag every question toward the same two pages, and two-letter terms like "KV" were being dropped. The fix is IDF weighting: rare words count more than common ones. After the fix, the KV cache question retrieved the KV cache guide. 2. **Ordering failure.** With excerpts before the question, the model missed a number that was verbatim in its context and wrongly abstained. Putting the question first fixed it. Small models read long contexts worse than short ones, so tell them what to look for before showing them where to look. The behavior we kept is the one support bots are judged on: across every run, the model either answered from the excerpts or abstained. It never invented a number. The abstain-when-unsure failure mode costs a resolution; the invent-when-unsure failure mode costs your credibility. ## Reuse the pattern The whole agent is three static files: the docs bundle ([/llms-full.txt](https://yourwildcard.ai/llms-full.txt)) that the site already generates for coding agents, a web worker that loads the ONNX model over WebGPU, and a page that does keyword retrieval in a few dozen lines of JavaScript. No framework, no vector database, no server. If your docs already ship an llms-full.txt, you are one worker file away from the same feature. ## Next steps - [Use the chat agent](/chat) and watch the tokens per second it reports. - [Take Inference The Hard Way](https://chaiwithjai.github.io/inference-the-hard-way/) to learn the budgeting used here, and [Kubernetes The Hard Way](https://bootstrap-your-own-k8s.netlify.app) to learn what the server-side option would have required. See something wrong? [Fix it](https://yourwildcard.ai/docs/contribute/contributor-guide.md). --- --- title: 'Ternary Bonsai 27B field guide' description: 'Run PrismML Ternary Bonsai 27B locally, understand its ternary package, reproduce measured M4 Pro results, and follow the primary sources.' status: published last_reviewed: 2026-07-15 canonical_url: https://yourwildcard.ai/docs/prismml/bonsai-27b --- # Bonsai 27B field guide Ternary Bonsai 27B is the main model covered by these docs. This guide connects the exact public checkpoint, the whitepaper, working local commands, measured M4 Pro traces, and 26 short lessons about what we learned. ## Start with the exact artifact The model is derived from Qwen3.6 27B. PrismML represents the language weights with the ternary values minus one, zero, and plus one, plus group scales. The ideal language-weight representation is about 5.9 GB. The current public MLX package is 8.49 GB because packaging, scales, biases, and the vision tower add bytes. - [Bonsai 27B Hugging Face collection](https://huggingface.co/collections/prism-ml/bonsai-27b) - [Ternary Bonsai 27B MLX 2-bit checkpoint](https://huggingface.co/prism-ml/Ternary-Bonsai-27B-mlx-2bit) - [Unpacked checkpoint](https://huggingface.co/prism-ml/Ternary-Bonsai-27B-unpacked) - [Bonsai 27B whitepaper](https://github.com/PrismML-Eng/Bonsai-demo/blob/main/bonsai-27b-whitepaper.pdf) - [PrismML announcement](https://prismml.com/news/bonsai-27b) Our traced file came from revision `70f75f3ad081ab840a42f3304c02c27e7f89bfb7`. Its `model.safetensors` file was 8,490,785,104 bytes with SHA-256 `8acd4597893ea7004e2d7336c3cf6e3157b8896592bbcf066db004021e45846b`. ## Run it on an Apple silicon Mac Install MLX LM and start its OpenAI-compatible server: ```bash uv tool install mlx-lm mlx_lm.server \ --model prism-ml/Ternary-Bonsai-27B-mlx-2bit \ --host 127.0.0.1 \ --port 8080 ``` LM Studio 0.4.19 also loaded the checkpoint on our test Mac. Its local API reported the model as `ternary-bonsai-27b-mlx`. A 24 GB machine is enough for moderate text work with one loaded runtime and a controlled 4K context. Long context, vision input, other large applications, or a second model server reduce the margin. ## Read the measured local result The controlled July 14 run used LM Studio, MLX, an M4 Pro Mac with 24 GB unified memory, and a 4,096 token context. LM Studio reported a 9.03 second load and 7.94 GiB loaded runtime size. | Measurement | Controlled run | Repeat run | | -------------------------- | -------------: | -----------: | | Median time to first token | 14.506 s | 13.383 s | | Median decode rate | 21.55 tok/s | 22.67 tok/s | | Median prompt processing | 101.06 tok/s | 109.54 tok/s | | Prompt tokens per run | 1,466 | 1,466 | | Output tokens per run | 127 | 127 | Both runs completed a structured `get_weather` tool call with `{"city":"Lisbon"}` and the `tool_calls` finish reason. This checks one response shape. It does not prove that long agent loops, recovery, or tool choice are reliable. ## Budget reasoning tokens deliberately The same local server behaved differently at two reasoning budgets: - At 16 completion tokens, it spent 15 tokens reasoning and returned no final answer. - At 256 completion tokens, it spent 191 tokens reasoning and returned `42`. That is an application concern, not a trivia detail. A model can consume a small output budget before it reaches the user-facing answer. Test the reasoning and answer budgets together. ## Keep comparisons honest PrismML reports 18.0 generation tokens per second on an M4 Pro using llama.cpp Metal. Our local runs used LM Studio and MLX, with different prompts and harness details. They show that interactive local use is practical. They do not prove that MLX is faster than PrismML's llama.cpp path. For reproducible comparisons, save the checkpoint revision and hash, runtime version, context limit, prompt and output token counts, cache state, model reload state, and every result, including slow runs. ## Read the lessons The [Bonsai 27B section of the blog](/blog) contains 26 focused answers. Start with these: - [What is Ternary Bonsai 27B?](/blog/what-is-ternary-bonsai-27b) - [How to run Ternary Bonsai 27B with MLX](/blog/run-ternary-bonsai-27b-mlx) - [What we learned running it on a 24 GB Mac](/blog/ternary-bonsai-27b-24gb-memory) - [How we benchmarked the model](/blog/ternary-bonsai-27b-benchmark-method) - [Why reasoning-token budgets matter](/blog/ternary-bonsai-27b-reasoning-token-budget) - [What the ecosystem is saying](/blog/bonsai-27b-ecosystem-reactions) ## Follow the evidence The full trace and profiling notes live in [Inference The Hard Way](https://github.com/ChaiWithJai/inference-the-hard-way/tree/main/profiles/ternary-bonsai-27b). The [ecosystem roundup](/blog/bonsai-27b-ecosystem-reactions) links directly to The Information, PrismML's launch thread, Khosla team amplification, Together AI, NVIDIA RTX Spark, Ion Stoica, and other public reactions. It labels company claims, investor support, reporting, and local measurements separately. --- --- title: PrismML Model Family status: published audience: DevRel owner: PrismML liaison source_tier: primary_links benchmark_status: source_only last_reviewed: 2026-07-07 canonical_url: https://yourwildcard.ai/docs/prismml/model-family --- # Model family PrismML is a company that builds small compressed language and image models, released under the Bonsai name, that you can run on a laptop, a phone, or in a browser. On this page you can learn what each Bonsai family is, see the licenses and file sizes we have confirmed against official sources, and pick a model to run yourself. ## Understand PrismML's pitch PrismML describes its work as "concentrating intelligence". The claim is that denser models need less memory and energy and are easier to deploy. Speed, memory reduction, and energy numbers on PrismML pages are vendor claims until we run our own benchmarks, so treat them as the vendor's statements, not confirmed results. The "world's first 1-bit AI model" line is a marketing claim. The body of the press page narrows it to "commercially viable" 1-bit models, so quote it with attribution only. PrismML's work spans most of the supply chain. The company designs the compressed formats (1-bit and ternary weights in GGUF and MLX files, defined in [weights on disk](https://yourwildcard.ai/docs/technical-guides/weights-on-disk.md)), maintains a llama.cpp fork with custom kernels that unpack the compressed weights during execution, and ships end products such as browser demos and the Bonsai Studio iPhone app. ## Pick a model family PrismML ships three Bonsai families. For example, if you want a text model on a Mac, download the 1-bit Bonsai 8B GGUF file (1.16 GB) and run it with llama.cpp by following the [Bonsai on llama.cpp recipe](https://yourwildcard.ai/docs/build-and-run/bonsai-llamacpp.md). | Family | What it is | Official sources | | --- | --- | --- | | 1-bit Bonsai | Compressed text model family for local and edge inference. | PrismML site, Bonsai release pages, Hugging Face collection, Bonsai demo repo, PrismML llama.cpp fork. | | Ternary Bonsai | Text model family that trades some compression for quality by using ternary weights. | PrismML release page and Hugging Face collection. | | Bonsai Image 4B | Local image generation model for browser, GPU, and iOS demos. | PrismML release page, Hugging Face collection, Bonsai Image demo repo, Bonsai Studio app page. | ## Check the facts we verified We confirmed these items against official PrismML sources. Full rows live in the [claim/source matrix](https://yourwildcard.ai/docs/prismml/claim-source-matrix.md). - The Bonsai 8B GGUF release is licensed Apache 2.0. The model card at https://huggingface.co/prism-ml/Bonsai-8B-gguf states apache-2.0 and the repo includes a LICENSE file. - The 1-bit GGUF file Bonsai-8B-Q1_0.gguf is 1.16 GB in the Hugging Face file listing at https://huggingface.co/prism-ml/Bonsai-8B-gguf/tree/main. - Bonsai Image 4B ships in 1-bit and ternary variants under Apache 2.0 per the release post at https://prismml.com/news/bonsai-image-4b. - The release post lists 0.93 GB for the 1-bit variant and 1.21 GB for the ternary variant. Both figures cover the diffusion transformer only. The full deployment payload on Apple Silicon is 3.42 GB for 1-bit and 3.88 GB for ternary, so always state which size you mean. - A 1-bit MLX build exists at https://huggingface.co/prism-ml/Bonsai-8B-mlx-1bit under Apache 2.0. The card lists a 1.28 GB model based on Qwen3-8B. - Bonsai Studio is a PrismML iOS app for running Bonsai Image 4B on iPhone. The PrismML home page links to the App Store listing (Bonsai Studio by PrismML, id 6767042620). We checked these facts on 2026-07-06. Some Bonsai product names circulate only in social media posts. We do not repeat a product name here until a primary source confirms it. When you cite a claim, keep PrismML's own statements separate from third-party interpretation. ## Next steps - [Run Bonsai with llama.cpp](https://yourwildcard.ai/docs/build-and-run/bonsai-llamacpp.md) to try the 1-bit text model on your own machine. - [Run Bonsai with MLX](https://yourwildcard.ai/docs/build-and-run/bonsai-mlx.md) if you prefer the Apple Silicon path. - [Trace every claim in the claim/source matrix](https://yourwildcard.ai/docs/prismml/claim-source-matrix.md) before you quote a PrismML number. See something wrong? [Fix it](https://yourwildcard.ai/docs/contribute/contributor-guide.md). --- --- title: Whitepaper Benchmarks status: published audience: DevRel owner: Docs librarian source_tier: primary benchmark_status: published last_reviewed: 2026-07-07 canonical_url: https://yourwildcard.ai/docs/prismml/whitepaper-benchmarks --- # Whitepaper benchmarks The 1-bit Bonsai 8B whitepaper reports how a 1.15 GB model scores against full-precision models ten times its size, and how fast and efficiently it runs on real hardware. On this page you will learn what each benchmark measures, what the paper found, and how to reproduce the numbers yourself. Every number on this page comes from the paper: [1-bit Bonsai 8B: End-to-end 1-bit language model deployment](https://yourwildcard.ai/whitepaper/1-bit-bonsai-8b-whitepaper.pdf) (PrismML, March 31, 2026). Read it alongside this page; this page is the plain-English tour. ## See the headline result The paper's central claim is about the tradeoff between model size and capability. Bonsai 8B keeps 70.5 average benchmark score while shrinking from 16.38 GB (FP16) to 1.15 GB on disk. That moves the score-per-gigabyte frontier, as the scatter below shows. ## Understand the six benchmarks The paper evaluates six skill categories. If you explain Bonsai to a developer, these are the six things the scores actually mean. | Benchmark | What it measures | Bonsai 8B | Qwen 3 8B (16.38 GB) | | --- | --- | --- | --- | | MMLU-Redux | Knowledge: multiple-choice questions across school and professional subjects | 65.7 | 83 | | MuSR | Reasoning: multi-step problems told as short stories | 50 | 55 | | GSM8K | Math: grade-school word problems solved step by step | 88 | 93 | | HumanEval+ | Coding: write a Python function that passes hidden tests | 73.8 | 82.3 | | IFEval | Instruction following: obey precise formatting rules | 79.8 | 81.5 | | BFCLv3 | Tool calling: produce a correct function call | 65.7 | 81 | Read the table honestly: Bonsai 8B gives up points against its own full-precision base model, most visibly on knowledge (MMLU-Redux) and tool calling (BFCLv3). What it buys with those points is a model 1/14 the size that beats or matches several 16 GB models overall (it outscores LFM2 8B, Llama 3.1 8B, GLM 4 9B, and Hermes 3 8B on average). ## Explain intelligence density The paper's most useful concept for the ecosystem is intelligence density: how much capability a model delivers per gigabyte. The paper defines it as the model's error exponent, D = -log(Pe) / N, where Pe treats the average benchmark score as a success probability and N is the size in GB. The definition rewards the hard climb at the top of the scale, because moving from 90 to 99 is worth more than moving from 50 to 55. By this measure the three Bonsai models lead every model in the 1.2B to 9B comparison, and not by a little: | Model | Intelligence density (1/GB) | Size | Average score | | --- | --- | --- | --- | | 1-bit Bonsai 1.7B | 2.832 | 0.24 GB | 49.6 | | 1-bit Bonsai 4B | 1.744 | 0.57 GB | 62.7 | | 1-bit Bonsai 8B | 1.060 | 1.15 GB | 70.5 | | Qwen 3 0.6B (next best) | 0.549 | 1.19 GB | 48.0 | | Qwen 3 8B | 0.096 | 16.38 GB | 79.3 | ## Read the speed and energy numbers The paper reports throughput as tg128 (tokens per second while generating 128 tokens) and pp512 (tokens per second while processing a 512-token prompt). The speedups compare against the FP16 build of the same model on the same machine. | Platform | Backend | Generation (tok/s) | Speedup vs FP16 | | --- | --- | --- | --- | | RTX 4090 | llama.cpp CUDA | 368 | 6.2x | | Mac M4 Pro | MLX | 131 | 8.4x | | Mac M4 Pro | llama.cpp Metal | 85 | 5.4x | | RTX 3060 Laptop | llama.cpp CUDA | 81 | 23x, because FP16 does not fit in its VRAM | | iPhone 17 Pro Max | MLX Swift | 44 | 3.2x vs 4-bit, because FP16 does not fit on the phone | Energy per generated token drops 4x to 6x on Mac and desktop GPU (for example 0.074 mWh/token vs 0.415 on M4 Pro with MLX). The paper is explicit that the iPhone energy figure is estimated from Xcode Power Profiler and battery drain, not hardware-metered. Quote it with that caveat. 1-bit weights do not make the math easier; they make the memory traffic smaller. Generation speed at batch size 1 is limited by how fast weights stream through memory, so a 14x smaller weight file means fewer bytes per token. The [bandwidth ledger guide](https://yourwildcard.ai/docs/technical-guides/bandwidth-ledger.md) walks through this arithmetic. ## Check what you need to reproduce them Pick the environment that matches what you want to verify. | Environment | What you can verify | What you need | | --- | --- | --- | | Your laptop or desktop | Throughput, memory, offline behavior | Apple Silicon Mac (16 GB RAM comfortable) or an NVIDIA GPU; about 2 GB disk for the [1.15 GB GGUF](https://huggingface.co/prism-ml/Bonsai-8B-gguf); the [llama.cpp recipe](https://yourwildcard.ai/docs/build-and-run/bonsai-llamacpp.md) | | A notebook (Colab or Jupyter) | Single benchmarks, e.g. GSM8K, on a rented GPU | A GPU runtime, [EvalScope](https://github.com/modelscope/evalscope), and the model card's [eval results folder](https://huggingface.co/prism-ml/Bonsai-8B-gguf/tree/main/.eval_results) to compare against | | Cloud GPUs | The paper's full evaluation matrix | The paper used EvalScope v1.4.2 with the vLLM 0.15.1 backend on NVIDIA H100s, greedy decoding at temperature 0, thinking mode disabled | The paper's methodology appendix lists the exact scoring rules per benchmark (rule-based first, LLM recall fallback only when parsing fails) and the fairness guarantees: identical infrastructure, generation parameters, dataset revisions, and judge configuration for all twelve models. ## Quote the numbers without overclaiming Three habits keep you accurate when you present these results: - Name the baseline. "8.4x faster" means versus FP16 of the same model on the same M4 Pro, not versus every other 8B model. - Keep quality and speed claims separate. The speed table does not say Bonsai matches Qwen 3 8B in quality; the benchmark table shows exactly where it does not. - Say which numbers are estimated. The iPhone energy figure and anything sourced from social posts stay labeled as estimates until measured. ## Next steps - [Run Bonsai on your machine](https://yourwildcard.ai/docs/build-and-run/bonsai-llamacpp.md) and compare your tokens per second against the paper's platform table. - [Record what you measure](https://yourwildcard.ai/docs/benchmarks/device-report-schema.md) so the ecosystem gets one more comparable data point. - [Read the whitepaper itself](https://yourwildcard.ai/whitepaper/1-bit-bonsai-8b-whitepaper.pdf) for the methodology appendices this page summarizes. See something wrong? [Fix it](https://yourwildcard.ai/docs/contribute/contributor-guide.md). --- --- title: Claim Source Matrix status: published audience: Docs librarian owner: Source reviewer source_tier: primary_links benchmark_status: source_only last_reviewed: 2026-07-07 canonical_url: https://yourwildcard.ai/docs/prismml/claim-source-matrix --- # Claim and source matrix The claim source matrix is the list of facts we publish about PrismML models, each traced to a named source with a verification status. Before you quote a number or a claim about PrismML anywhere on this site, look it up here to see whether it has a verified source. Here is how a row works. The claim "the 1-bit Bonsai 8B GGUF file is 1.16 GB" links to the official Hugging Face file listing, where a reviewer read the listed size of the file on 2026-07-06. Early discovery notes said about 1.15 GB, and the listing shows 1.16 GB, so 1.16 GB is the figure to use in prose. That is the whole method. A claim gets a source, a reviewer reads the source, and the row records what the source actually says. Each row carries one of these statuses: - **verified.** An official source states the fact directly and a reviewer confirmed it on the source, or we have run and benchmark evidence. - **partial.** A source exists, but it does not cover the full claim, so quote it with the caveat in the notes. - **source-linked.** A durable source exists, but a reviewer still needs the exact quote or version. - **discovery.** The signal comes only from social posts or search, so do not treat it as fact. - **quarantined.** Do not publish this as fact. We checked every row against its source on 2026-07-06. No benchmark has been run locally, so speed, memory, and energy numbers stay out of this table until we have logs. ## Look up a claim | Claim | Status | Source | What the source shows | | --- | --- | --- | --- | | The Bonsai 8B GGUF release is licensed Apache 2.0. | verified | [Hugging Face model card](https://huggingface.co/prism-ml/Bonsai-8B-gguf) | The license field on the card states apache-2.0, and the repo carries a 10.2 kB LICENSE file. | | The 1-bit Bonsai 8B GGUF file is 1.16 GB. | verified | [Hugging Face file listing](https://huggingface.co/prism-ml/Bonsai-8B-gguf/tree/main) | The listing shows Bonsai-8B-Q1_0.gguf at 1.16 GB. Use 1.16 GB in prose, not the 1.15 GB from early notes. | | Bonsai Image 4B is released in 1-bit and ternary variants under Apache 2.0. | verified | [PrismML release post](https://prismml.com/news/bonsai-image-4b) | The post names both variants and states the Apache 2.0 license for the release. | | Ternary Bonsai Image 4B is a 1.21 GB model. | partial | [PrismML release post](https://prismml.com/news/bonsai-image-4b) | The post confirms 1.21 GB, but that figure covers the diffusion transformer only. The full deployment payload on Apple Silicon is 3.88 GB. Always state which size you mean. | | 1-bit Bonsai Image 4B is a 0.93 GB model. | partial | [PrismML release post](https://prismml.com/news/bonsai-image-4b) | The post confirms 0.93 GB for the diffusion transformer only. The full deployment payload on Apple Silicon is 3.42 GB. We have not yet confirmed the figure on a Hugging Face file listing. | | Bonsai Studio is a PrismML iPhone app for on-device image generation. | verified | [PrismML home page](https://prismml.com/) | The home page links to the App Store listing for Bonsai Studio by PrismML (id 6767042620). The Bonsai Image 4B post calls it the iOS app for running Bonsai Image 4B on iPhone and says both variants run on-device on iPhone 17 Pro Max. | | PrismML launched the world's first 1-bit AI model. | quarantined | [PrismML press page](https://prismml.com/news/prismml-launches-worlds-first-1-bit-ai-model) | The headline says world's first, but the body narrows it to the "world's first commercially viable 1-bit large language models". This is a vendor marketing claim. Quote it with attribution and the qualifier, and do not publish it as an independent fact. | | A 1-bit MLX build of Bonsai 8B exists. | verified | [Hugging Face model card](https://huggingface.co/prism-ml/Bonsai-8B-mlx-1bit) | The repo exists under Apache 2.0 in 1-bit g128 MLX format at 1.28 GB, based on Qwen3-8B. The throughput numbers on the card are vendor claims, and we have not run them locally. | ## Claims you cannot quote yet Two claim areas have no row, so do not publish them as fact: - Memory reduction, speedup, and energy claims. The home page numbers (14x smaller, 8x faster, 5x more energy efficient) are vendor marketing until someone runs a local benchmark and files the logs. - Device support beyond iPhone. Each device needs its own row that separates official support from a demo someone got working once. ## Turn a social signal into a claim A post on X can point you at something real, e.g., a local demo, WebGPU interest, or edge deployment discussion. It is never a source on its own. When you find a claim on X, look for the official source first, add a row here with what that source says, and only then quote the claim in public prose. If no official source exists, the row stays at discovery status. The X findings that started this table came through that path, and the ones with official sources are now rows above. The raw findings live in the [X discovery notes](https://yourwildcard.ai/docs/sources/x-discovery-notes.md). ## Check a row before you add or quote it Ask these questions: - Does the row name the exact model, file, runtime, device, and date? - Is the source official or reproducible? - Is the claim scoped to a specific benchmark phase rather than a generic speed statement? - If the claim came from a social post, is that post treated as a pointer and not as the source? - Does the public page that quotes the claim match the row's status? ## Next steps - Read the [source policy](https://yourwildcard.ai/docs/sources/source-policy.md) for the rules behind these statuses. - See where raw signals come from in the [X discovery notes](https://yourwildcard.ai/docs/sources/x-discovery-notes.md). - Ready to add a row? Start with the [contributor guide](https://yourwildcard.ai/docs/contribute/contributor-guide.md). --- --- title: Supply Chain Layers status: published audience: DevRel owner: Ecosystem librarian source_tier: mixed benchmark_status: not_applicable last_reviewed: 2026-07-07 canonical_url: https://yourwildcard.ai/docs/ecosystem/supply-chain-layers --- # Supply-chain layers The ML supply chain is the chain of tools and decisions that take a small model from raw data to a product running on a real device. On this page you can see the nine layers of that chain, learn what question each layer answers, and find the tools that live in each one. ## Understand the nine layers Every tool in this ecosystem sits in one layer of the chain. For example, llama.cpp is a runtime. It answers the question "how does inference run on my machine?", and it sits downstream of the compression layer that produced the GGUF file it loads. When you know which layer a tool belongs to, you know what job it does and which tools sit next to it. | Layer | Question it answers | Tools and players in this layer | | --- | --- | --- | | Problems and users | What work should the model do, and how do we judge success? | Domain owners, benchmark authors, Arena-style evaluators. | | Data and features | What raw data, features, embeddings, prompts, and labels feed the system? | Feature stores, data pipelines, vector stores, Ray Data-style pipelines. | | Model architecture | Which backbone fits the workload? | Transformers, state space models, hybrid models, Liquid-style models, PrismML Bonsai. | | Training and post-training | How is the model adapted or aligned? | Unsloth, TRL, OpenRLHF, Axolotl, LLaMA-Factory. | | Compression and formats | How does the model fit into memory and onto devices? | GGUF, MLX, AWQ/GPTQ, FP8/MXFP4/NVFP4, PrismML 1-bit/ternary. | | Kernels and compilers | What makes the math fast enough? | CUDA, Triton, CUTLASS, FlashAttention, Metal, MLX kernels. | | Runtime and serving | How does inference run reliably and cheaply? | llama.cpp, MLX, Ollama, LM Studio, vLLM, SGLang, Ray Serve. | | Apps, agents, devices | Where does the model run for users? | Phones, laptops, browsers, local agents, robots, enterprise apps. | | Evaluation and feedback | How do we know it works and improves? | Human preference workflows, Arena-style evals, regression suites. | ## Watch where the layers pull against each other The layers are not independent. A choice in one layer changes what is possible in the next, and the tradeoffs below come up again and again when you read benchmark claims or plan a local deployment. - **Compression vs quality.** Smaller weights need less memory, but you still have to measure what the compression did to quality and task scope. - **Weight size vs KV cache.** With 1-bit weights, the KV cache can become the largest use of memory at long context. - **Prefill vs decode.** A speedup can apply to generation but not prompt processing, so check which phase a claim measures. - **Runtime support vs model release.** A model is not broadly usable until the runtimes people actually run, e.g., llama.cpp or Ollama, support it. - **Social excitement vs evidence.** Social posts are useful for discovering a tool, but treat their numbers as unconfirmed until a documented run backs them up. ## Next steps - Run a model through the runtime layer yourself with the [Bonsai on llama.cpp recipe](https://yourwildcard.ai/docs/build-and-run/bonsai-llamacpp.md). - Then read the [KV cache guide](https://yourwildcard.ai/docs/technical-guides/kv-cache.md) to understand the memory tradeoff you just saw in action. See something wrong? [Fix it](https://yourwildcard.ai/docs/contribute/contributor-guide.md). --- --- title: llama.cpp status: published audience: DevRel owner: Runtime validator source_tier: primary_links benchmark_status: not_run last_reviewed: 2026-07-07 canonical_url: https://yourwildcard.ai/docs/ecosystem/llama-cpp --- # llama.cpp llama.cpp is an open source C/C++ runtime that runs GGUF model files on your own computer, on CPU, NVIDIA CUDA, or Apple Metal. On this page you can learn what it does, see how to build and run it with Bonsai 8B, and find the full recipe. ## What it is llama.cpp is maintained at [ggml-org/llama.cpp](https://github.com/ggml-org/llama.cpp). Three things make it the first runtime to try: - It reads GGUF model files, which is the format PrismML uses for compressed releases such as [Bonsai 8B GGUF](https://huggingface.co/prism-ml/Bonsai-8B-gguf). - It builds with plain CMake on macOS, Linux, and Windows, so you can go from clone to prompt on one machine. - It ships a command line tool (`llama-cli`) and a local server (`llama-server`) that speaks the same HTTP API as OpenAI, so apps built for that API can talk to a model on your machine. llama.cpp serves one user on one device. If you need to serve many users at once, use vLLM or SGLang instead. PrismML maintains a fork at [PrismML-Eng/llama.cpp](https://github.com/PrismML-Eng/llama.cpp). The Bonsai instructions in this wiki build from that fork, not from upstream. We do not yet know whether upstream llama.cpp can load the Bonsai 1-bit GGUF files, so stay on the fork until someone records a successful upstream run. ## Run Bonsai with it The shortest path is to clone the fork, build it, and run one prompt. These commands come from the [Bonsai with llama.cpp recipe](https://yourwildcard.ai/docs/build-and-run/bonsai-llamacpp.md) and match the upstream build docs, but we have not yet recorded a run on hardware. ```bash git clone https://github.com/PrismML-Eng/llama.cpp cd llama.cpp cmake -B build # add -DGGML_CUDA=ON on NVIDIA hardware cmake --build build -j ./build/bin/llama-cli -m Bonsai-8B-Q1_0.gguf \ -p "Explain quantum computing in simple terms." -n 256 -ngl 99 ``` Download `Bonsai-8B-Q1_0.gguf` from the [official model card](https://huggingface.co/prism-ml/Bonsai-8B-gguf) before you run the last command. On macOS the build uses Metal by default, so you do not need any extra flags. You know it worked when the model streams an answer to your prompt. For a stronger check, turn networking off after the model download and run the prompt again. If it still answers, the model is running on your machine. The recipe uses the sampling flags `--temp 0.5 --top-p 0.85 --top-k 20`. We have not yet confirmed these against the model card, so treat them as a starting point. Do not silently swap the PrismML fork for upstream. Record which repo and commit you built, because Bonsai quantization support may differ between them. ## Serve it as a local API `llama-server` exposes the model over a local HTTP endpoint, so you can point an OpenAI client library at your own machine. We have not yet verified which routes the server provides, so check the server docs in the repo before you build on it. The [recipe](https://yourwildcard.ai/docs/build-and-run/bonsai-llamacpp.md) includes a `llama-server` path. If you want a packaged experience instead of building from source, Ollama and LM Studio run the same GGUF files. ## What we have not verified We list open items plainly so you know what to trust: - No one has recorded a build and run on hardware yet. The commands above are checked against the upstream docs only. - We do not know whether upstream llama.cpp loads Bonsai 1-bit GGUF files or whether the fork is required. - We have not read the llama.cpp license or the Bonsai 8B GGUF weight license, so this page makes no license claims. - We have not listed which quantization files besides `Bonsai-8B-Q1_0.gguf` exist on the model card. Third-party posts on X describe PrismML as focused on edge and single-user inference efficiency. That framing is unconfirmed, so rely on the official repos and the model card instead. ## Sources | URL | What it shows | | --- | --- | | [ggml-org/llama.cpp](https://github.com/ggml-org/llama.cpp) | Upstream project, build system, CLI and server binaries, GGUF support. | | [PrismML-Eng/llama.cpp](https://github.com/PrismML-Eng/llama.cpp) | Fork used by the current Bonsai build path. | | [prism-ml/Bonsai-8B-gguf](https://huggingface.co/prism-ml/Bonsai-8B-gguf) | Official GGUF release of Bonsai 8B, the exact model filename, and the license. | | [PrismML-Eng/Bonsai-demo](https://github.com/PrismML-Eng/Bonsai-demo/) | Official PrismML demo pairing Bonsai with a local runtime. | ## Next steps - [Run Bonsai with llama.cpp](https://yourwildcard.ai/docs/build-and-run/bonsai-llamacpp.md) walks the full path with an environment record and success checks. - [KV cache](https://yourwildcard.ai/docs/technical-guides/kv-cache.md) explains what the runtime does with memory while it generates tokens. See something wrong? [Fix it](https://yourwildcard.ai/docs/contribute/contributor-guide.md). --- --- title: MLX (Apple) status: published audience: DevRel engineer owner: Runtime validator source_tier: primary_links benchmark_status: not_run last_reviewed: 2026-07-07 canonical_url: https://yourwildcard.ai/docs/ecosystem/mlx --- # MLX MLX is Apple's open source machine learning framework for Apple Silicon, and it is the native way to run PrismML Bonsai models on a Mac. On this page you can learn what MLX does, find the Bonsai model files it needs, and decide between MLX and the llama.cpp path. ## Understand what MLX gives you Apple's machine learning research team publishes MLX as an official open source project ([project page](https://opensource.apple.com/projects/mlx)). The [official docs](https://ml-explore.github.io/mlx/) describe its design: - An array API similar to NumPy, with higher level packages for neural networks and optimizers. - Lazy computation with dynamic graph construction. - Unified memory. Arrays live in shared memory, so operations can run on the CPU or the GPU without explicit data transfers. - Front ends in Python, C++, and Swift. For running a compressed model like Bonsai 8B 1-bit, the useful part is that MLX is a first party Mac runtime. You do not need CUDA or a translation layer, and you do not manage GPU offload flags the way you do on NVIDIA hardware, because the CPU and GPU share the same memory. ## Get the model files PrismML publishes a dedicated MLX artifact for Bonsai 8B at [Bonsai 8B MLX 1-bit](https://huggingface.co/prism-ml/Bonsai-8B-mlx-1bit). This is a separate file from the [GGUF release](https://huggingface.co/prism-ml/Bonsai-8B-gguf) that llama.cpp uses. Each runtime reads its own format, so a model ships as one artifact per runtime rather than one universal file. Before you install, check the model card for two things we have not yet confirmed: which macOS versions and Apple Silicon chips the artifact supports, and whether loading it requires mlx-lm or PrismML specific code. We also do not yet know how the MLX artifact compares to the GGUF file in disk size or runtime memory. We have not run Bonsai 8B on MLX yet. The links here come from the official docs and the model card. Take install commands, speed figures, and memory figures from those sources or from a completed device report, not from this page. ## Choose between MLX and llama.cpp MLX runs only on Apple Silicon. llama.cpp runs on most platforms, uses the GGUF format, and has a working recipe at [Bonsai with llama.cpp](https://yourwildcard.ai/docs/build-and-run/bonsai-llamacpp.md). Ollama and LM Studio also run GGUF models on a Mac. If you only have a Mac and want the first party path, use MLX. If you want one setup that works across machines, use llama.cpp. If you want to compare the two, run the same Bonsai 8B prompt set on MLX and on llama.cpp with Metal on the same machine, and record time to first token, tokens per second, and peak memory. Record the chip, the RAM, and the macOS version with your numbers, and confirm the model still runs offline after the download. Posts on X associate PrismML models with edge deployment and single request inference, which is the workload MLX on a laptop serves. Those posts are unconfirmed, so do not cite them as claims. ## Next steps - Run the model yourself with the [Bonsai with MLX](https://yourwildcard.ai/docs/build-and-run/bonsai-mlx.md) recipe. - Prefer a cross platform setup? Follow [Bonsai with llama.cpp](https://yourwildcard.ai/docs/build-and-run/bonsai-llamacpp.md). See something wrong? [Fix it](https://yourwildcard.ai/docs/contribute/contributor-guide.md). --- --- title: Ollama status: published audience: DevRel engineer owner: Runtime validator source_tier: primary_links benchmark_status: not_run last_reviewed: 2026-07-07 canonical_url: https://yourwildcard.ai/docs/ecosystem/ollama --- # Ollama Ollama is a tool that downloads, runs, and serves language models on your own laptop or workstation with one command. On this page you can learn what Ollama does, how it compares to llama.cpp, MLX, and LM Studio, and what we know so far about running Bonsai on it. ## Understand what Ollama does Ollama is a local model runtime and manager published at [ollama.com](https://ollama.com/). It exposes a local HTTP API that applications call to generate text, chat, and manage models on your machine, per the [official API introduction](https://docs.ollama.com/api/introduction). Three properties make it useful for local work with small models: - One tool covers download, run, and serve, so you do one step instead of three. - The API stays up as a local endpoint, so agents, scripts, and benchmark harnesses can call it instead of running a CLI command for each prompt. - It loads packaged model files in the GGUF format (defined in [weights on disk](https://yourwildcard.ai/docs/technical-guides/weights-on-disk.md)), so it can run whatever quantized files the compression tools produce. We only cite Ollama's official site and API docs here. Claims beyond those two pages, e.g., which GPU backends and operating systems it supports, are left out until we can point at the official page that states them. ## Compare it with the other local runtimes Ollama's peers are [llama.cpp](https://yourwildcard.ai/docs/build-and-run/bonsai-llamacpp.md), which you build from source for the most control, MLX, which is native to Apple Silicon, and LM Studio, which is a GUI app. Ollama is easier to script than a GUI and needs less setup than a source build, so it is the fastest way to get a model answering prompts locally. It also connects to the layers around it. Upstream, it consumes quantized model files, e.g., the [Bonsai 8B GGUF](https://huggingface.co/prism-ml/Bonsai-8B-gguf) from the official model card, and it inherits inference kernels from its underlying engine, so kernel improvements show up as tokens per second and memory changes. Downstream, local agents, editors, and demos call its API, and you can point a benchmark script at the same endpoint to measure time to first token, tokens per second, and peak memory, then compare against llama.cpp and MLX on the same machine. ## Know what is verified and what is not Nobody has recorded a Bonsai run on Ollama on real hardware. Do not present `ollama run` with a Bonsai model as working in talks or docs until someone runs it and records the device, model file, command, and results. Two things are still unknown: whether stock Ollama loads Bonsai 8B's 1-bit GGUF quantization or needs the PrismML llama.cpp fork's kernels, and whether the local API is compatible with the OpenAI API format. Everything above comes from these official sources: - [ollama.com](https://ollama.com/), the product home. - [docs.ollama.com/api/introduction](https://docs.ollama.com/api/introduction), the API documentation. - [huggingface.co/prism-ml/Bonsai-8B-gguf](https://huggingface.co/prism-ml/Bonsai-8B-gguf), the Bonsai 8B GGUF model card, which is the file an Ollama setup would load. Research notes from X place Ollama in the contributor conversation around local runtime recipes for compressed models. That is unconfirmed, so treat any specific compatibility or performance claim from those threads as a rumor until an official page states it. ## Next steps - Try the [Bonsai with Ollama recipe](https://yourwildcard.ai/docs/build-and-run/bonsai-ollama.md). It is a draft, so expect gaps. - Run the verified path first with the [Bonsai on llama.cpp recipe](https://yourwildcard.ai/docs/build-and-run/bonsai-llamacpp.md). See something wrong? [Fix it](https://yourwildcard.ai/docs/contribute/contributor-guide.md). --- --- title: LM Studio status: published audience: DevRel engineer owner: Runtime validator source_tier: primary_links benchmark_status: not_run last_reviewed: 2026-07-07 canonical_url: https://yourwildcard.ai/docs/ecosystem/lm-studio --- # LM Studio LM Studio is a desktop app that downloads and runs language models on your own machine through a graphical interface, with no command line. On this page you can learn where LM Studio fits among the local runtimes, walk through a run of PrismML Bonsai in it, and see which facts are still unconfirmed. ## Know where it fits LM Studio runs packaged model files such as [Bonsai 8B GGUF](https://huggingface.co/prism-ml/Bonsai-8B-gguf). It sits in the same layer as llama.cpp, MLX, and Ollama, and the difference is the interface. With [llama.cpp](https://yourwildcard.ai/docs/build-and-run/bonsai-llamacpp.md) you build the runtime yourself and work in a terminal. With LM Studio you install an app, pick a model, and chat with it. That makes it the easiest way to check that a model runs on a device offline, and the best demo tool for an audience that will not compile anything. A working LM Studio setup is also the base for offline demos and privacy checklists, and any run you record with it can feed the device benchmark data (device, runtime, model, quantization, tokens per second, time to first token, memory). The URLs on this page come from the official LM Studio site and are the only claims we treat as confirmed. We have not yet checked feature details such as supported formats, server modes, OS requirements, or license terms against [the app docs](https://lmstudio.ai/docs/app), and we have not yet run Bonsai in LM Studio on a recorded device. ## Run Bonsai in LM Studio Here is the run we want you to be able to reproduce. We have not yet completed it on a recorded device, so treat it as a checklist rather than a verified recipe. 1. Install LM Studio from [lmstudio.ai](https://lmstudio.ai/) and note the app version. 2. Download the [Bonsai 8B GGUF](https://huggingface.co/prism-ml/Bonsai-8B-gguf) file from the official model card. 3. Load the model, disconnect the network, and run one prompt. 4. Record your device, OS, app version, model file, quantization, output, time to first token, tokens per second, and peak memory. The [llama.cpp recipe](https://yourwildcard.ai/docs/build-and-run/bonsai-llamacpp.md) shows the environment record format. 5. Export or screenshot the model and load settings so someone else can reproduce your run. If you use the same model file as the llama.cpp path, your numbers show the overhead of the app itself. For a live talk, a flow that works is: install the app, load Bonsai 8B GGUF, disconnect the network, run one prompt, and show the memory footprint. ## Check what is still unconfirmed Before you rely on LM Studio for a workshop or a commercial demo, confirm these points against [the app docs](https://lmstudio.ai/docs/app), because we have not: - whether it loads PrismML 1-bit and ternary GGUF files, not just standard quantizations - whether it exposes a local OpenAI-compatible server, which you would need to mirror the llama.cpp server setup - the supported operating systems and minimum hardware - the license terms for workshop and commercial use - what the app sends over the network after the model download, which decides what "offline" means One more caution. Our research pass on X (2026-07-06) mentioned LM Studio only as part of a general trend toward better local tooling, not in any PrismML post. So do not claim that PrismML models run in LM Studio today until you have loaded one yourself. ## Official sources - [lmstudio.ai](https://lmstudio.ai/) is the product home. - [lmstudio.ai/docs/app](https://lmstudio.ai/docs/app) is the app documentation and the source to quote for any feature, format, or OS claim. - [Bonsai 8B GGUF on Hugging Face](https://huggingface.co/prism-ml/Bonsai-8B-gguf) is the model card for the file to load. - [llama.cpp on GitHub](https://github.com/ggml-org/llama.cpp) is the home of the GGUF format that LM Studio consumes. ## Next steps - [Run Bonsai 8B in LM Studio](https://yourwildcard.ai/docs/build-and-run/bonsai-lmstudio.md) walks through the run step by step. - [Supply chain layers](https://yourwildcard.ai/docs/ecosystem/supply-chain-layers.md) shows where runtimes sit in the wider toolchain. See something wrong? [Fix it](https://yourwildcard.ai/docs/contribute/contributor-guide.md). --- --- title: vLLM status: published audience: DevRel engineer owner: Runtime validator source_tier: primary_links benchmark_status: not_run last_reviewed: 2026-07-07 canonical_url: https://yourwildcard.ai/docs/ecosystem/vllm --- # vLLM vLLM is an open source engine that serves a language model checkpoint as a high-throughput, OpenAI-compatible API. On this page you can learn what makes it fast, see where it sits among the other tools in this map, and find the primary sources to cite when you explain it. ## Understand what vLLM does vLLM came out of the Berkeley systems ecosystem, the same group behind Ray, SGLang, and Chatbot Arena. You give it a model checkpoint, and it exposes an inference server that many clients can call at once. Its speed comes from a few core ideas, all documented in the [Anatomy of vLLM post](https://blog.vllm.ai/2025/09/05/anatomy-of-vllm.html) and the [official docs](https://docs.vllm.ai/): - **PagedAttention.** The engine stores KV cache entries in fixed-size blocks and allocates and frees the blocks on demand, so long and variable-length sequences do not fragment GPU memory. - **Continuous batching.** New requests join a running batch at each scheduling step instead of waiting for the current batch to finish, so the GPU stays busy. - **Prefix caching.** Requests that share a prompt prefix reuse the same KV blocks, so repeated prompts start faster. - **OpenAI-compatible server.** vLLM exposes the same API that llama.cpp's `llama-server` exposes locally, so you can swap backends without rewriting clients. If terms like KV cache are new to you, read [the KV cache guide](https://yourwildcard.ai/docs/technical-guides/kv-cache.md) first. It explains why the cache takes most of the serving memory, and vLLM's paging design is the reference example it builds on. ## Place it among the other tools vLLM sits in the runtime and serving layer, between the model formats it consumes and the apps that call it. One example of a full handoff: a quantized checkpoint in AWQ format arrives from the compression layer, vLLM loads it and runs attention kernels from the FlashAttention lineage, and an agent framework downstream calls the resulting endpoint. | Neighbor | Direction | Handoff | | --- | --- | --- | | Compression and formats | upstream | Quantized checkpoints (AWQ/GPTQ, FP8 family) arrive as inputs. Check runtime compatibility for each format rather than assuming it. | | Kernels and compilers | upstream | vLLM consumes attention kernels (FlashAttention lineage) and adds KV paging on top. | | Apps, agents, and devices | downstream | Apps and agents call the OpenAI-compatible endpoint it serves. | | SGLang and Ray Serve | peers | SGLang is the same layer with a different cache strategy (RadixAttention/HiCache). Ray Serve orchestrates distributed serving around engines like vLLM. | vLLM is the wrong tool for the phone, laptop, and browser deployments PrismML targets. For local paths use [Bonsai with llama.cpp](https://yourwildcard.ai/docs/build-and-run/bonsai-llamacpp.md). This page covers vLLM because the cost of serving any model, compressed or not, comes down to the same KV cache concepts. ## Cite it without overclaiming When you write or talk about vLLM, cite these sources: | URL | What it evidences | | --- | --- | | [https://docs.vllm.ai/](https://docs.vllm.ai/) | The official documentation: install paths, serving API, supported models and quantization formats. | | [https://blog.vllm.ai/2025/09/05/anatomy-of-vllm.html](https://blog.vllm.ai/2025/09/05/anatomy-of-vllm.html) | The official engineering walkthrough of the engine internals: KV paging, scheduling, batching. | | [https://www.lmsys.org/blog/2024-01-17-sglang/](https://www.lmsys.org/blog/2024-01-17-sglang/) | The SGLang side of the cache strategy comparison (RadixAttention). | Everything on this page comes from the upstream docs. We have not yet installed vLLM or served a model on our own hardware, so do not publish throughput or latency numbers for vLLM from memory or social posts. We also have not found a primary source that says whether vLLM supports PrismML's Bonsai formats (1-bit/ternary), so treat Bonsai serving as local-only for now. ## Next steps - Read [the KV cache guide](https://yourwildcard.ai/docs/technical-guides/kv-cache.md) to understand the memory problem PagedAttention solves. - See [the supply chain layers](https://yourwildcard.ai/docs/ecosystem/supply-chain-layers.md) for where runtime and serving fits in the full map. - Run [Bonsai with llama.cpp](https://yourwildcard.ai/docs/build-and-run/bonsai-llamacpp.md) to try the local counterpart of what vLLM does in the datacenter. - These engines run on Kubernetes in production. Build that layer yourself in [Kubernetes The Hard Way](https://yourwildcard.ai/docs/learning-paths/kubernetes-the-hard-way.md). See something wrong? [Fix it](https://yourwildcard.ai/docs/contribute/contributor-guide.md). --- --- title: SGLang status: published audience: DevRel engineer owner: Runtime validator source_tier: primary_links benchmark_status: not_run last_reviewed: 2026-07-07 canonical_url: https://yourwildcard.ai/docs/ecosystem/sglang --- # SGLang SGLang is an open source server for running large language models on GPUs at high throughput, built by the LMSYS team at Berkeley. On this page you can learn what makes its caching design different, see how it fits next to vLLM and llama.cpp, and find the official install path. ## What SGLang does When a model server handles a prompt, it first computes an internal state called the KV cache for every token in the prompt. That step is called prefill, and it is expensive. SGLang is built around reusing that work instead of repeating it. Two design ideas define it: - **RadixAttention** ([LMSYS blog post](https://www.lmsys.org/blog/2024-01-17-sglang/)). SGLang keeps KV cache entries in a radix tree keyed by token sequences. When a new request shares a prefix with an earlier one, e.g., the same system prompt, few-shot examples, or chat history, SGLang reuses the cached state for that prefix automatically instead of recomputing it. - **HiCache** ([design doc](https://docs.sglang.ai/advanced_features/hicache_design.html)). The KV cache extends past GPU memory into host memory and storage. A reusable prefix that gets evicted from the GPU can come back from a slower tier instead of being recomputed from scratch. For example, an agent that sends the same 2,000-token system prompt with every request only pays the prefill cost for that prompt once. Every later request starts from the cached state and only computes the new tokens. The LMSYS launch post quotes speedup multipliers, but those numbers apply to the workloads and hardware they measured. We do not repeat them here because we have not reproduced them. ## Know when to reach for it SGLang serves an OpenAI-compatible endpoint, so it slots in where you would otherwise put vLLM. The useful contrasts: - **vLLM** is the peer GPU server. Its PagedAttention manages KV cache memory in fixed-size blocks. RadixAttention is a different idea, and it is about reuse across requests rather than memory layout. Neither approach is simply better, and this page only claims the mechanism difference. - **llama.cpp, MLX, Ollama, and LM Studio** are the local single-device paths. Pick them for laptops and small machines, and pick SGLang or vLLM when you have a GPU server and many concurrent requests. To try the local path, see [Run Bonsai with llama.cpp](https://yourwildcard.ai/docs/build-and-run/bonsai-llamacpp.md). - SGLang consumes checkpoints and quantized formats from upstream. Whether PrismML Bonsai 1-bit and ternary formats load in SGLang is unconfirmed. Check the [Bonsai 8B GGUF model card](https://huggingface.co/prism-ml/Bonsai-8B-gguf) against SGLang's supported formats before you plan a demo on it. - RadixAttention manages the cache. It is not a new attention kernel, and SGLang still runs on top of standard attention kernels. Workloads with heavy prefix sharing benefit most, e.g., multi-turn chat, agent loops, and batch evaluation over a fixed prompt template. ## Install and verify it yourself We have not yet run SGLang on hardware in this project. Start from the official install instructions at [https://docs.sglang.ai/](https://docs.sglang.ai/) rather than copying commands from an older tutorial, because server flags and install extras change between releases. A quick way to confirm the prefix cache is working once you have a server up: 1. Launch the server with a small open model and note the exact command, model ID, and GPU. 2. Send the same long-prefix prompt twice through the OpenAI-compatible endpoint. 3. Compare the time to first token for both requests. The second should be faster because the prefix comes from cache. We have not confirmed SGLang's license or minimum GPU requirement here. Check the official docs for both before you commit to it. Any throughput, speedup, or cache hit rate figure for SGLang must come from a run you recorded or cite the exact source URL it was taken from. Multipliers in blog posts are specific to the workloads they measured. ## Sources | URL | What it covers | | --- | --- | | [https://docs.sglang.ai/](https://docs.sglang.ai/) | Official docs, with the install path, server usage, and supported features | | [https://www.lmsys.org/blog/2024-01-17-sglang/](https://www.lmsys.org/blog/2024-01-17-sglang/) | RadixAttention design and launch claims | | [https://docs.sglang.ai/advanced_features/hicache_design.html](https://docs.sglang.ai/advanced_features/hicache_design.html) | HiCache hierarchical KV cache architecture | ## Next steps - Read [how the KV cache works](https://yourwildcard.ai/docs/technical-guides/kv-cache.md) to understand what RadixAttention is reusing. - See [where runtimes sit in the supply chain](https://yourwildcard.ai/docs/ecosystem/supply-chain-layers.md) to place SGLang among its neighbors. - Run a model locally first with the [Bonsai llama.cpp recipe](https://yourwildcard.ai/docs/build-and-run/bonsai-llamacpp.md) if you do not have a GPU server. - These engines run on Kubernetes in production. Build that layer yourself in [Kubernetes The Hard Way](https://yourwildcard.ai/docs/learning-paths/kubernetes-the-hard-way.md). See something wrong? [Fix it](https://yourwildcard.ai/docs/contribute/contributor-guide.md). --- --- title: Ray / Anyscale status: published audience: DevRel engineer owner: Recipe author source_tier: primary_links benchmark_status: not_run last_reviewed: 2026-07-07 canonical_url: https://yourwildcard.ai/docs/ecosystem/ray --- # Ray and Anyscale Ray is an open source Python framework for running data processing, model training, and model serving across many machines. On this page you can see where Ray fits in the supply chain and try its install command. ## What Ray is Ray came out of the Berkeley systems group around Ion Stoica ([Ion Stoica home page](https://people.eecs.berkeley.edu/~istoica/), [Ray](https://www.ray.io/)). You write normal Python, and Ray spreads the work across a cluster. It ships three libraries that each cover one layer of the supply chain: - **Ray Data** ([docs](https://docs.ray.io/en/latest/data/data.html)) handles the data and features layer. It reads, transforms, and writes datasets, and it feeds training jobs and batch inference. - **Ray Train** ([docs](https://docs.ray.io/)) handles the training and post-training layer. It runs multi-node training jobs alongside fine-tuning stacks such as Unsloth, TRL, OpenRLHF, and LLaMA-Factory. - **Ray Serve** ([docs](https://docs.ray.io/)) handles the runtime and serving layer. It sits above inference engines such as vLLM and SGLang and handles distribution and autoscaling. It does not replace those engines. Anyscale is the company behind Ray and sells a managed Ray platform. We have not yet captured an Anyscale product page as a source, so this page makes no claims about the Anyscale platform. Stoica's framing of the ecosystem has three parts, which are data, training, and serving ([Columbia Engineering interview](https://www.engineering.columbia.edu/about/news/ion-stoica-highlights-open-source-efficiency-across-ai-stack)). Ray Data, Ray Train, and Ray Serve map onto those three parts one to one. When you explain the ecosystem in a talk, Ray is the one project that lets you point at all three layers at once. ## Decide whether you need Ray Ray is cluster-scale infrastructure. If you run a small model on one machine with llama.cpp, MLX, Ollama, or LM Studio, you do not need Ray. Start with the [Bonsai on llama.cpp recipe](https://yourwildcard.ai/docs/build-and-run/bonsai-llamacpp.md) instead. Ray becomes useful when you move from one device to batch data processing, multi-node training, or serving that scales across machines. ## Try Ray This is the minimal install path. These commands come from the upstream docs, and we have not yet run them on hardware. ```bash pip install -U "ray[data,train,serve]" python -c "import ray; ray.init(); print(ray.cluster_resources())" ``` If the second command prints a dictionary of CPUs and memory, Ray started a local cluster on your machine. See the [Ray docs](https://docs.ray.io/) for what to run next, such as a Ray Data batch job that reads, transforms, and writes a dataset. ## Sources Every claim on this page comes from one of these pages: - [ray.io](https://www.ray.io/), the official project site. - [docs.ray.io](https://docs.ray.io/), the official docs, which cover the install path, Ray Train, and Ray Serve. - [Ray Data docs](https://docs.ray.io/en/latest/data/data.html), which describe Ray Data as the AI data processing layer. - [Ion Stoica's home page](https://people.eecs.berkeley.edu/~istoica/) and a [Columbia Engineering interview](https://www.engineering.columbia.edu/about/news/ion-stoica-highlights-open-source-efficiency-across-ai-stack), which cover his role in the Berkeley systems ecosystem and his open source efficiency argument. We have not yet recorded Ray's license, current release version, or the details of the Ray Serve and vLLM integration from primary sources, so this page does not state them. ## Next steps - [Supply-chain layers](https://yourwildcard.ai/docs/ecosystem/supply-chain-layers.md) shows the full map that Ray's three libraries slot into. - [Run Bonsai with llama.cpp](https://yourwildcard.ai/docs/build-and-run/bonsai-llamacpp.md) is the single-machine path if you do not need a cluster yet. - These engines run on Kubernetes in production. Build that layer yourself in [Kubernetes The Hard Way](https://yourwildcard.ai/docs/learning-paths/kubernetes-the-hard-way.md). See something wrong? [Fix it](https://yourwildcard.ai/docs/contribute/contributor-guide.md). --- --- title: Unsloth status: published audience: DevRel engineer owner: Recipe author source_tier: primary_links benchmark_status: not_run last_reviewed: 2026-07-07 canonical_url: https://yourwildcard.ai/docs/ecosystem/unsloth --- # Unsloth Unsloth is a toolkit for fine-tuning language models, which means training an existing model further on your own data so it behaves the way you want. On this page you can learn where Unsloth fits in the path from a small model's weights to a running local product, and find the official docs to start a fine-tuning run of your own. ## What Unsloth does Unsloth adapts a model you already have. You give it a base model and a dataset, and it runs supervised fine-tuning (SFT) or a lighter method such as LoRA or QLoRA, which train a small set of extra weights instead of the whole model. The official documentation lives at [unsloth.ai/docs](https://unsloth.ai/docs), and the [fine-tuning guide](https://unsloth.ai/docs/get-started/fine-tuning-llms-guide) walks through a full run. Comparable tools include LLaMA-Factory, TRL, OpenRLHF, and Axolotl. TRL and OpenRLHF focus on preference alignment methods such as DPO and PPO, which you would run after or instead of an Unsloth SFT pass. We have not benchmarked these tools against each other, so we do not claim a winner. Unsloth also lists a product called Unsloth Studio, described as a local app for working with models. We have only seen this name in our own ecosystem notes and have not confirmed it against an Unsloth product page, so treat it as unconfirmed. The links on this page come from the official Unsloth docs, but we have not yet installed Unsloth or completed a fine-tuning run on our own hardware. Until we do, treat every step as upstream guidance rather than a tested recipe. ## See where fine-tuning fits Fine-tuning sits between preparing data and shipping a model to a local runtime: ```text data and features -> training and post-training <- Unsloth lives here -> compression and formats (GGUF/MLX export targets) -> runtime and serving (llama.cpp, Ollama, LM Studio, MLX) -> evaluation and feedback (regression evals after tuning) ``` In practice the flow looks like this. You format a dataset, fine-tune a small model with Unsloth, export the tuned weights to a local runtime format such as GGUF or MLX, run the model with a tool like llama.cpp, and then run evaluations to check that tuning did not hurt quality. The [Bonsai llama.cpp recipe](https://yourwildcard.ai/docs/build-and-run/bonsai-llamacpp.md) shows the running step of that flow. One open question for this ecosystem: PrismML Bonsai models use 1-bit and ternary weight formats, and we do not yet know which quantized bases Unsloth supports for training or whether its output feeds that compression path. We have asked the Unsloth team and have no answer yet. ## Plan a fine-tuning run If you want to try Unsloth yourself, follow the [official fine-tuning guide](https://unsloth.ai/docs/get-started/fine-tuning-llms-guide). A run you could later share back here would: 1. Install Unsloth from a named release or commit, per the official docs. 2. Fine-tune one named small model on a documented dataset. 3. Record the device, VRAM use, wall-clock time, and final loss or eval score. 4. Export the tuned model to a local runtime format and complete one prompt, following the pattern in the [Bonsai llama.cpp recipe](https://yourwildcard.ai/docs/build-and-run/bonsai-llamacpp.md). After a verified run, before-and-after quality comparisons and device benchmark rows (tokens per second, time to first token, RAM and VRAM) become possible for the tuned model. ## Next steps - Read [supply chain layers](https://yourwildcard.ai/docs/ecosystem/supply-chain-layers.md) to see how the training layer connects to the rest of the ecosystem. - Run the [Bonsai llama.cpp recipe](https://yourwildcard.ai/docs/build-and-run/bonsai-llamacpp.md) to try the runtime step that a tuned model would flow into. See something wrong? [Fix it](https://yourwildcard.ai/docs/contribute/contributor-guide.md). --- --- title: Liquid AI (LFMs) status: published audience: DevRel engineer owner: Docs librarian source_tier: primary_links benchmark_status: not_run last_reviewed: 2026-07-07 canonical_url: https://yourwildcard.ai/docs/ecosystem/liquid-ai --- # Liquid AI Liquid Foundation Models (LFMs) are a family of small language models from Liquid AI that do not use the standard transformer design, and Liquid builds them to run on small devices such as laptops and phones. On this page you can find the official LFM docs and see how LFMs compare with PrismML Bonsai. ## Find the official material Liquid AI documents the LFM family in its [getting started docs](https://docs.liquid.ai/lfm/getting-started/welcome). The [complete model library](https://docs.liquid.ai/lfm/models/complete-library) lists every model, and the LFM2 generation has a public [technical report on arXiv](https://arxiv.org/html/2511.23404v1). We have not yet pulled parameter counts, context lengths, licenses, or file formats out of those pages, so this entry does not state any. When you need one of those numbers, quote it from the model library or the LFM2 report and keep the URL. Check the [model library](https://docs.liquid.ai/lfm/models/complete-library) for three things per model: whether the weights are open and under what license, which file formats it ships in (e.g., GGUF or MLX), and which local runtimes such as llama.cpp, Ollama, or LM Studio can load it. We have not run any LFM on hardware yet, so this page has no benchmark numbers. ## Understand where LFMs fit Transformers are still the default design for language models. State space and hybrid models such as Mamba and the LFMs are attempts to do better on long context, memory use, and device performance. For example, an LFM aims to answer a long prompt on a laptop without filling its memory, where a transformer of the same size might struggle. LFMs sit at the model architecture layer of the supply chain. Once you know which formats a given LFM ships in, the next questions move down the chain: which runtimes load it, and how it behaves on a device. ## Compare LFMs with Bonsai LFMs and PrismML Bonsai chase the same goal from different directions. Both want useful output per GB of memory, per watt, and per dollar on small devices. - Bonsai keeps a known transformer architecture and pushes compression to 1-bit and ternary weights. - LFMs change the architecture itself to cut compute and memory cost. Because the two take different paths, a fair comparison must hold the runtime, the device, and the quantization constant. Comparing parameter counts alone tells you little. Any published claim about LFM performance, such as long context or on device speed, should come from the [LFM2 report](https://arxiv.org/html/2511.23404v1) with a section reference. We have not run a head to head test between an LFM and Bonsai yet. ## Next steps - Read [supply chain layers](https://yourwildcard.ai/docs/ecosystem/supply-chain-layers.md) to see where model architecture sits among the other layers. - Run [Bonsai with llama.cpp](https://yourwildcard.ai/docs/build-and-run/bonsai-llamacpp.md) to get a working local baseline you could later compare an LFM against. See something wrong? [Fix it](https://yourwildcard.ai/docs/contribute/contributor-guide.md). --- --- title: Runtime Map status: published audience: DevRel engineer owner: Runtime validator source_tier: mixed benchmark_status: not_run last_reviewed: 2026-07-07 canonical_url: https://yourwildcard.ai/docs/build-and-run/runtime-map --- # Runtime map A runtime is the program that loads a PrismML model's weights and generates text on your hardware, e.g., llama.cpp on a laptop. Use this page to pick the runtime that fits your machine and jump to the steps for running it. The most reliable path today is llama.cpp. It runs GGUF model files as a local command line tool or a small local server, and we have a written recipe for it that comes from the upstream docs. We have not yet run that recipe on hardware, so treat the steps as source-checked rather than proven. If you are on a Mac, MLX is the Apple Silicon path, and the [Bonsai with MLX recipe](https://yourwildcard.ai/docs/build-and-run/bonsai-mlx.md) is a draft that has not been run on hardware yet. Draft recipes also exist for [Ollama](https://yourwildcard.ai/docs/build-and-run/bonsai-ollama.md) and [LM Studio](https://yourwildcard.ai/docs/build-and-run/bonsai-lmstudio.md). ## Pick a runtime Each row tells you whether you can run a PrismML model there today. | Runtime | What it is | Can you run PrismML on it today? | | --- | --- | --- | | llama.cpp | Local command line tool and server for GGUF models. | Yes. Follow the [Bonsai on llama.cpp recipe](https://yourwildcard.ai/docs/build-and-run/bonsai-llamacpp.md). The commands come from the upstream docs; we have not yet run them on hardware. | | MLX | Apple's framework for running models on Apple Silicon. | Probably. Follow the [Bonsai with MLX recipe](https://yourwildcard.ai/docs/build-and-run/bonsai-mlx.md). It is a draft, and we have not run it on hardware. | | Ollama | Local model runner with a simple install and pull workflow. | Unknown. The [Bonsai with Ollama recipe](https://yourwildcard.ai/docs/build-and-run/bonsai-ollama.md) is a draft, and we have not checked whether Ollama can load the current Bonsai GGUF files without conversion. | | LM Studio | Desktop app for downloading and chatting with local models. | Unknown. The [Bonsai with LM Studio recipe](https://yourwildcard.ai/docs/build-and-run/bonsai-lmstudio.md) is a draft, and we have not checked the import flow or model compatibility. | | Browser (WebGPU) | Running a model inside a web page using the GPU. | Not yet. We have only seen social and search mentions, and we have not found an official demo repo or model artifact. | | iOS (mlx-swift) | Running a model on an iPhone or iPad. | Not yet. We have only seen source references, with no device run recorded. | | vLLM / SGLang | Server-side engines for serving models to many users at once. | Not applicable for a laptop. We list them for context and have not confirmed PrismML compatibility. | ## How we rate the evidence When a row says "yes" or "not yet", that judgment rests on how much evidence we have. We rate each runtime on a five step scale. - **discovery.** We have only seen social or search mentions. - **source_checked.** Official docs or repos show a working path. - **locally_run.** Someone ran one prompt and recorded the environment. - **benchmarked.** Someone captured standardized latency, throughput, memory, and task results. - **maintained.** The page has an owner and recent passing evidence. Right now llama.cpp sits at source_checked and every other runtime sits at discovery or below. No runtime on this page has benchmark numbers yet, so this page makes no speed or memory claims. This scale rates the evidence for a recipe, and the [device report schema](https://yourwildcard.ai/docs/benchmarks/device-report-schema.md) uses its own benchmark_status values for a recorded run. The two roughly line up, so locally_run matches a benchmark_status of partial, and benchmarked matches reproducible. ## Next steps - [Run Bonsai on llama.cpp](https://yourwildcard.ai/docs/build-and-run/bonsai-llamacpp.md). This is the most reliable recipe today, and the commands are checked against the upstream docs. - [Run Bonsai with MLX](https://yourwildcard.ai/docs/build-and-run/bonsai-mlx.md) if you are on an Apple Silicon Mac. The recipe is a draft and has not been run on hardware. - [Run Bonsai with Ollama](https://yourwildcard.ai/docs/build-and-run/bonsai-ollama.md). The recipe is a draft and has not been run on hardware. - [Run Bonsai with LM Studio](https://yourwildcard.ai/docs/build-and-run/bonsai-lmstudio.md). The recipe is a draft and has not been run on hardware. See something wrong? [Fix it](https://yourwildcard.ai/docs/contribute/contributor-guide.md). --- --- title: Bonsai With llama.cpp status: verified audience: DevRel engineer owner: Runtime validator source_tier: primary_links benchmark_status: partial last_reviewed: 2026-07-07 canonical_url: https://yourwildcard.ai/docs/build-and-run/bonsai-llamacpp --- # Bonsai with llama.cpp Bonsai is PrismML's 8B open-weight language model, and llama.cpp is a C++ runtime that runs models like it on a laptop or a single GPU. On this page you can build llama.cpp, load the Bonsai GGUF file, and run your first prompt locally. This path has been run on hardware once. On an Apple M4 Pro with 24 GB of RAM, the 1.16 GB `Bonsai-8B-Q1_0.gguf` file loaded in the prebuilt Homebrew `llama-cli` (build b9590) and generated a coherent answer at 65 tokens per second. The full record is at the bottom of this page. ## Get the model and the runtime You need two things: the Bonsai model file in GGUF format, and a build of PrismML's llama.cpp fork. - [Bonsai 8B GGUF model card](https://huggingface.co/prism-ml/Bonsai-8B-gguf), which hosts the model file, e.g., `Bonsai-8B-Q1_0.gguf` - [PrismML llama.cpp fork](https://github.com/PrismML-Eng/llama.cpp), the runtime you will build - [PrismML Bonsai collection](https://huggingface.co/collections/prism-ml/bonsai) and the [Bonsai demo repo](https://github.com/PrismML-Eng/Bonsai-demo/), if you want the full model family and example code Download the GGUF file from the model card before you build, and note the exact filename. The commands below assume it sits in the llama.cpp directory. ## Build and run on an NVIDIA GPU (CUDA) ```bash git clone https://github.com/PrismML-Eng/llama.cpp cd llama.cpp cmake -B build -DGGML_CUDA=ON cmake --build build -j ./build/bin/llama-cli \ -m Bonsai-8B-Q1_0.gguf \ -p "Explain quantum computing in simple terms." \ -n 256 \ --temp 0.5 \ --top-p 0.85 \ --top-k 20 \ -ngl 99 ``` If the build succeeds and the model loads, you should see the prompt followed by generated text in your terminal. That is your success check. ## Build and run on a Mac (Metal) The default macOS build uses Metal, so no extra flag is needed. ```bash git clone https://github.com/PrismML-Eng/llama.cpp cd llama.cpp cmake -B build cmake --build build -j ./build/bin/llama-cli \ -m Bonsai-8B-Q1_0.gguf \ -p "Explain quantum computing in simple terms." \ -n 256 \ --temp 0.5 \ --top-p 0.85 \ --top-k 20 \ -ngl 99 ``` ## Serve Bonsai as a local API Once the CLI run works, you can start a local server that speaks the OpenAI-compatible API, so any client library can call your machine. ```bash ./build/bin/llama-server \ -m Bonsai-8B-Q1_0.gguf \ --host 0.0.0.0 \ --port 8080 \ -ngl 99 ``` After the first model download, this whole path can run offline. We have not confirmed that on a device yet. ## Fix common failures If something breaks, it is usually one of these three problems. | Symptom | Likely cause | Fix | | --- | --- | --- | | Build fails | Upstream build instructions changed | Pull the latest fork and re-run cmake | | Model does not load | Wrong file or unsupported quantization | Re-check the exact filename on the model card | | Low throughput | GPU path not enabled | Rebuild with the CUDA flag, or check `-ngl` is set | ## Record what you measured If you note the tokens per second, time to first token, and peak memory from your run, you can share numbers that others can compare against. The [device report schema](https://yourwildcard.ai/docs/benchmarks/device-report-schema.md) shows which fields to capture. ## Compare against a recorded run One run of this recipe has been recorded so far. It was a short single prompt, not the paper's standardized tg128/pp512 protocol, so compare direction rather than exact numbers. The whitepaper reports 85 tokens per second for llama.cpp Metal on an M4 Pro with 48 GB; this run measured 65 tokens per second on the 24 GB M4 Pro. ```markdown - Date: 2026-07-06 - Device: Mac (Apple M4 Pro, 24 GB RAM), macOS 26.5 - Runtime: llama.cpp b9590 (Homebrew llama-cli, Metal) - Model file: Bonsai-8B-Q1_0.gguf, 1,158,654,496 bytes (1.16 GB) - Prompt: "Explain in two sentences why the KV cache grows with context length." - Settings: -n 128, --single-turn, --seed 42 - Prompt processing: 140 tokens/s - Generation: 65 tokens/s - Peak memory: not measured - Output: coherent two-sentence answer, on topic ``` ## Next steps - [Understand the KV cache](https://yourwildcard.ai/docs/technical-guides/kv-cache.md), the mechanism behind the memory and speed numbers you just saw - [Pick another runtime from the runtime map](https://yourwildcard.ai/docs/build-and-run/runtime-map.md) to compare Bonsai across runtimes See something wrong? [Fix it](https://yourwildcard.ai/docs/contribute/contributor-guide.md). --- --- title: Bonsai With MLX status: published audience: DevRel engineer owner: Recipe author source_tier: primary_links benchmark_status: not_run last_reviewed: 2026-07-07 canonical_url: https://yourwildcard.ai/docs/build-and-run/bonsai-mlx --- # Bonsai with MLX MLX is Apple's machine learning framework for Apple Silicon Macs, and Bonsai 8B is PrismML's small open-weight language model. On this page you install MLX, download the official 1-bit Bonsai 8B weights, run one prompt on your Mac, and record what you measure. These commands come from the upstream MLX docs and the Bonsai model card links; we have not yet run them on a Mac. Steps we could not confirm are marked "inferred". Do not cite this page as proof that Bonsai 8B runs under MLX. ## Check what you need You need an Apple Silicon Mac (any M-series chip). MLX does not run on Intel Macs. You also need Python 3 and enough free disk space and unified memory for the model. We have not confirmed the minimum memory, so read it off the [model card](https://huggingface.co/prism-ml/Bonsai-8B-mlx-1bit) before you download, along with the license. | You need | Details | Source | | --- | --- | --- | | Device | Apple Silicon Mac (M-series) | [MLX project page](https://opensource.apple.com/projects/mlx) | | Runtime | MLX, installed below | [MLX docs](https://ml-explore.github.io/mlx/) | | Model | 1-bit Bonsai 8B MLX weights from the official PrismML org | [Model card](https://huggingface.co/prism-ml/Bonsai-8B-mlx-1bit) | For background on the model itself, see the [Bonsai 8B announcement](https://prismml.com/news/bonsai-8b). The model card lists the 1-bit MLX build at 1.28 GB, and that figure has a verified row in the [claim/source matrix](https://yourwildcard.ai/docs/prismml/claim-source-matrix.md). ## Install MLX and download the model Step 1. Install MLX in a fresh virtual environment, per the [official docs](https://ml-explore.github.io/mlx/). ```bash python3 -m venv .venv && source .venv/bin/activate pip install mlx ``` Step 2. Install `mlx-lm` and download the model. This step is inferred. The MLX docs cover the core framework, and `mlx-lm` is the companion package people commonly use to run MLX-format Hugging Face checkpoints. Check the model card for the officially supported load path first, because the 1-bit checkpoint may need a PrismML-specific loader. ```bash pip install mlx-lm hf download prism-ml/Bonsai-8B-mlx-1bit ``` ## Run one prompt This command is inferred. If the model card documents its own run command, use that instead. ```bash mlx_lm.generate \ --model prism-ml/Bonsai-8B-mlx-1bit \ --prompt "Explain quantum computing in simple terms." \ --max-tokens 256 ``` If the model responds, you have a working local run. After the model is cached, turn Wi-Fi off and run the same command again, and note whether generation still works offline. If the offline run fails, the loader is probably re-resolving the Hugging Face repo, so point the command at the local download path instead of the repo id. We do not yet know the recommended sampling settings for Bonsai under MLX. The [llama.cpp recipe](https://yourwildcard.ai/docs/build-and-run/bonsai-llamacpp.md) uses temp 0.5, top-p 0.85, and top-k 20, but we have not confirmed those carry over. ## Record what you measured Write down your numbers so someone else can reproduce the run. Capture: - **Tokens per second.** Use the number the runtime reports. `mlx_lm.generate` prints generation stats, and if it does not, time the run and divide. - **Time to first token.** The time from command start to the first generated token. - **Peak memory.** Unified memory used during generation, from Activity Monitor or the `mlx.core.metal` memory APIs in the MLX docs. - **Model file size.** The size on disk after download, plus the exact filenames. - **Offline behavior.** Whether the run worked with Wi-Fi off. Copy this block and fill it in as you go: ```markdown - Date: - Operator: - Device (chip, e.g. M3 Pro): - OS: - RAM (unified memory): - Runtime + version (mlx / mlx-lm): - Model: prism-ml/Bonsai-8B-mlx-1bit - Model file(s) + size on disk: - Command: - Prompt: - Output: - TTFT: - Tokens/sec: - Peak memory: - Offline mode verified: yes/no ``` ## Fix common problems If a step fails, the table below covers the failures we expect, based on how MLX and 1-bit checkpoints usually behave. When you hit one, note the exact error and your mlx and mlx-lm versions, because that is what the fix usually depends on. | Symptom | Likely cause | Fix | | --- | --- | --- | | `mlx_lm` cannot load the checkpoint | Stock `mlx-lm` may not support 1-bit quantization | Check the model card for a PrismML-specific load path or fork | | Install fails | Intel Mac or unsupported Python version | MLX needs Apple Silicon. Check your chip and Python version against the MLX docs | | Very low tokens per second | Memory pressure or swap on a low-RAM device | Close other apps, and record your RAM and peak memory alongside the number | | Works online, fails offline | Loader re-resolving the Hugging Face repo | Point the command at the local download path instead of the repo id | ## Next steps - [Record your run in a device report](https://yourwildcard.ai/docs/benchmarks/device-report-schema.md) so others can reproduce your numbers. - [Read the MLX ecosystem entry](https://yourwildcard.ai/docs/ecosystem/mlx.md) for what MLX is and where it fits. - [Try the llama.cpp recipe](https://yourwildcard.ai/docs/build-and-run/bonsai-llamacpp.md) to run Bonsai on non-Apple hardware. See something wrong? [Fix it](https://yourwildcard.ai/docs/contribute/contributor-guide.md). --- --- title: Bonsai 8B With Ollama status: published audience: DevRel engineer owner: Runtime validator source_tier: primary_links benchmark_status: not_run last_reviewed: 2026-07-07 canonical_url: https://yourwildcard.ai/docs/build-and-run/bonsai-ollama --- # Bonsai with Ollama Ollama is a free desktop tool that runs open-weight language models on your own machine, and Bonsai 8B is PrismML's small 1-bit quantized model. On this page you can import the Bonsai 8B GGUF file into Ollama, run one prompt from the command line and one through the local API, and record what happened. These commands come from the upstream Ollama docs; we have not yet run them against Bonsai 8B. Ollama documents the GGUF import path, but Bonsai's 1-bit quantization may not load in stock Ollama. The [llama.cpp recipe](https://yourwildcard.ai/docs/build-and-run/bonsai-llamacpp.md) uses a PrismML fork of llama.cpp, and Ollama ships its own copy of llama.cpp. So the first thing to test is whether the model loads at all. ## Check what you need You need a macOS, Linux, or Windows machine. A GPU is optional because Ollama falls back to CPU ([ollama.com](https://ollama.com/)). You also need the model file, `Bonsai-8B-Q1_0.gguf`, from the [Bonsai 8B GGUF model card](https://huggingface.co/prism-ml/Bonsai-8B-gguf). Confirm the exact filename and size on the model card. Expect to need roughly the file size plus context overhead in RAM. Posts on X cite "1.15 GB" for 1-bit Bonsai 8B, but that figure is unconfirmed and needs a primary source. Check the license on the model card before you redistribute anything. The Bonsai Image 4B release used Apache 2.0, but do not assume the same license applies to Bonsai 8B. There is no known `ollama pull` name for Bonsai 8B in the Ollama library. Until PrismML or a maintainer publishes one, the Modelfile import path below is the documented route. ## Import the model and run a prompt Steps 1 and 2 follow the official Ollama docs. The Modelfile parameters in step 3 come from the sampling flags in the [llama.cpp recipe](https://yourwildcard.ai/docs/build-and-run/bonsai-llamacpp.md), not from an Ollama-specific PrismML doc. 1. Install Ollama from [ollama.com](https://ollama.com/) and confirm the daemon runs with `ollama --version`. 2. Download the GGUF file from the [model card](https://huggingface.co/prism-ml/Bonsai-8B-gguf) and note its exact filename and size in bytes. 3. Write a Modelfile next to the downloaded file: ```text FROM ./Bonsai-8B-Q1_0.gguf PARAMETER temperature 0.5 PARAMETER top_p 0.85 PARAMETER top_k 20 ``` 4. Create and run the model: ```bash ollama create bonsai-8b -f Modelfile ollama run bonsai-8b "Explain quantum computing in simple terms." ``` If `ollama run` prints a sensible answer, the model loaded and you have a working local setup. 5. Call the local API ([API docs](https://docs.ollama.com/api/introduction)): ```bash curl http://localhost:11434/api/generate -d '{ "model": "bonsai-8b", "prompt": "Explain quantum computing in simple terms.", "stream": false }' ``` 6. Disconnect from the network and repeat step 4 to confirm the model runs offline. ## Record what you measured If the run works, write down enough detail that someone else can reproduce it. Copy this block and fill it in as you go: ```markdown - Date: - Operator: - Device: - OS: - CPU: - GPU: - RAM / VRAM: - Runtime: Ollama - Model file (name + size on disk): - Quantization / format: - Modelfile contents: - Command: - Prompt: - Output: - TTFT: - Tokens/sec (from `ollama run --verbose` or API `eval_count` / `eval_duration`): - Peak memory (RAM and VRAM if GPU used): - Offline mode verified: yes/no ``` You can get tokens per second and time to first token straight from the API response fields `eval_count`, `eval_duration`, and `prompt_eval_duration`, with no extra tooling ([API docs](https://docs.ollama.com/api/introduction)). ## Fix common problems If `ollama create` rejects the GGUF, Ollama's bundled llama.cpp probably does not support Bonsai's 1-bit quantization. Save the Ollama version and the full error text, then fall back to the [llama.cpp recipe](https://yourwildcard.ai/docs/build-and-run/bonsai-llamacpp.md), which uses PrismML's patched fork. If the model loads but the output is garbage, the Modelfile is probably missing the right chat template. Check the model card for a prompt template and add a `TEMPLATE` directive. If throughput is low, the model may be running on CPU only or not fitting in VRAM. Run `ollama ps` and check GPU utilization. If the API returns connection refused, the Ollama daemon is not running. Start the app or run `ollama serve`, and confirm it listens on port 11434. ## Sources - [Ollama](https://ollama.com/) - [Ollama API docs](https://docs.ollama.com/api/introduction) - [Bonsai 8B GGUF model card](https://huggingface.co/prism-ml/Bonsai-8B-gguf) - [PrismML Bonsai collection](https://huggingface.co/collections/prism-ml/bonsai) - [1-bit Bonsai 8B announcement](https://prismml.com/news/bonsai-8b) ## Next steps - [Record your run in a device report](https://yourwildcard.ai/docs/benchmarks/device-report-schema.md) so others can compare results. - [Read the Ollama ecosystem entry](https://yourwildcard.ai/docs/ecosystem/ollama.md) for background on the runtime itself. - [Try the llama.cpp recipe](https://yourwildcard.ai/docs/build-and-run/bonsai-llamacpp.md) if the GGUF does not load here. See something wrong? [Fix it](https://yourwildcard.ai/docs/contribute/contributor-guide.md). --- --- title: Bonsai With LM Studio status: published audience: DevRel engineer owner: Recipe author source_tier: primary_links benchmark_status: not_run last_reviewed: 2026-07-07 canonical_url: https://yourwildcard.ai/docs/build-and-run/bonsai-lmstudio --- # Bonsai with LM Studio LM Studio is a free desktop app that downloads and runs local language models behind a graphical chat interface, with no command line required. On this page you load Bonsai 8B, PrismML's [1-bit small model](https://prismml.com/news/bonsai-8b), into LM Studio, run a prompt with the network disconnected, and save your settings so someone else can repeat the run. These steps come from the official LM Studio and PrismML docs. We have not yet run them on a device. Where the docs do not spell out a step, we say so with the word "inferred". If you complete a run, please record what happened. ## Before you start You need four things: - A desktop machine. LM Studio runs on macOS, Windows, and Linux. Check the exact OS and hardware requirements in the [LM Studio app docs](https://lmstudio.ai/docs/app) before you start. - The latest LM Studio release from [lmstudio.ai](https://lmstudio.ai/). Note the version number, because you will record it later. - The Bonsai 8B GGUF file from the [model card](https://huggingface.co/prism-ml/Bonsai-8B-gguf). The llama.cpp recipe uses the filename `Bonsai-8B-Q1_0.gguf`, so confirm that name against the model card. - Enough disk space and RAM for the model. We do not have a confirmed file size yet. An [X post by Vinod Khosla](https://x.com/vkhosla/status/2039168431065563399) mentioned a 1.15 GB 1-bit Bonsai 8B, but that number is unconfirmed, so check the size on the model card. One risk to know about before you invest time. Bonsai uses [1-bit, also called ternary, quantization](https://prismml.com/news/ternary-bonsai), which means each weight is stored in about one bit instead of 16. PrismML maintains its own [fork of llama.cpp](https://github.com/PrismML-Eng/llama.cpp) to run these files, and LM Studio bundles the standard llama.cpp engine. The standard engine may or may not load the file. If it refuses, that is a useful result too, and the troubleshooting section below tells you what to capture. ## Run Bonsai in LM Studio 1. Download and install LM Studio from [lmstudio.ai](https://lmstudio.ai/). Write down the version string. 2. Get the model file. There are two paths: - Search for `prism-ml/Bonsai-8B-gguf` inside the app. This is inferred, because LM Studio's search reads from Hugging Face, so confirm it against [the app docs](https://lmstudio.ai/docs/app). - Or download the file from the [model card](https://huggingface.co/prism-ml/Bonsai-8B-gguf) in a browser and import it through the local models directory. The exact import path is also inferred, so confirm it in the docs. 3. Load the model in a chat session. If the engine rejects the 1-bit file, copy the exact error text and add it as a compatibility note to the [llama.cpp recipe](https://yourwildcard.ai/docs/build-and-run/bonsai-llamacpp.md). 4. Run one prompt. Use the same prompt as the llama.cpp recipe so the two runs can be compared: ```text Explain quantum computing in simple terms. ``` 5. Match the sampling settings from the [llama.cpp recipe](https://yourwildcard.ai/docs/build-and-run/bonsai-llamacpp.md) wherever the UI exposes them. Set temperature to 0.5, top-p to 0.85, top-k to 20, and max tokens to 256. 6. Check offline behavior. After the model has fully downloaded, disconnect from the network and run the prompt again. Note whether it works. 7. Export your settings. LM Studio stores configuration per model. Save it as a preset or config export and attach it to your results. The export mechanism is inferred, so confirm it in [the app docs](https://lmstudio.ai/docs/app). We also do not yet know what format the export takes or whether it moves cleanly between LM Studio versions, so note what you find. 8. Optionally, test the local server. If your LM Studio version exposes an OpenAI-compatible server, start it and run one `curl` completion. Note the host, port, and response. If you want to compare against a reference setup, the [PrismML Bonsai demo repo](https://github.com/PrismML-Eng/Bonsai-demo/) shows how PrismML runs the model. ## Record your results Copy this block before you start, fill it in as you go, and file it as a [device report](https://yourwildcard.ai/docs/benchmarks/device-report-schema.md). ```markdown - Date: - Operator: - Device: - OS: - CPU / GPU: - RAM / VRAM: - LM Studio version: - Model file and size on disk: - Quantization / format: - Prompt: - Output: - TTFT: - Tokens/sec (from LM Studio's stats readout): - Peak memory: - Offline mode verified: yes/no - Settings export attached: yes/no ``` ## Troubleshoot common failures The most likely failure is the model refusing to load, because of the 1-bit quantization risk described above. Whatever goes wrong, capture the exact error text and your LM Studio version, since those two facts usually decide whether the problem is the app or the file. | Symptom | Likely cause | What to try | | --- | --- | --- | | Model fails to load | The bundled llama.cpp engine lacks the 1-bit support that [PrismML's fork](https://github.com/PrismML-Eng/llama.cpp) has. | Check the engine version in the LM Studio release notes, then test the same file in the fork directly. Capture the engine version and the exact error text. | | Model not found in search | The GGUF repo is not indexed, or it was renamed. | Download it from the [model card](https://huggingface.co/prism-ml/Bonsai-8B-gguf) in a browser and import it manually. Note the search term you used and the filename. | | Offline run fails | The app is calling out to the network, or the model was not fully cached. | Re-test with the model fully downloaded before you disconnect. Note whether reconnecting fixes it. | | Throughput far below llama.cpp | GPU offload is disabled, or the engine defaults differ. | Check LM Studio's GPU offload setting and compare against a llama.cpp run with `-ngl 99` on the same device. Attach your settings export and both benchmark rows. | ## Next steps - [Record what you measured in a device report](https://yourwildcard.ai/docs/benchmarks/device-report-schema.md) - [Read the LM Studio ecosystem entry](https://yourwildcard.ai/docs/ecosystem/lm-studio.md) - [Compare with the llama.cpp recipe](https://yourwildcard.ai/docs/build-and-run/bonsai-llamacpp.md) See something wrong? [Fix it](https://yourwildcard.ai/docs/contribute/contributor-guide.md). --- --- title: Bonsai Image 4B Local Demo status: published audience: DevRel engineer owner: Recipe author source_tier: primary_links benchmark_status: not_run last_reviewed: 2026-07-07 canonical_url: https://yourwildcard.ai/docs/build-and-run/bonsai-image --- # Bonsai Image 4B local Bonsai Image 4B is PrismML's small text-to-image model, released in 1-bit and ternary variants under the Apache 2.0 license. This page gives you the official sources, the setup outline, and a measurement checklist, so you can clone the demo, follow its README to generate one image on your own machine, and record what you measured so someone else can reproduce your run. The commands below come from the official sources linked next to them. We have not yet run them on hardware. Steps marked inferred are our best reading of the upstream docs, so follow the demo README where the two disagree. ## Check what you need You need the official demo code and one model file. The demo lives at [PrismML-Eng/Bonsai-image-demo](https://github.com/PrismML-Eng/Bonsai-image-demo) on GitHub, and its README names the supported runtime, the install steps, and the minimum device requirements. The [Bonsai Image 4B announcement](https://prismml.com/news/bonsai-image-4b) describes the model itself. PrismML positions the model for phones, laptops, and browsers, but the exact RAM and OS requirements come from the README. The model ships in two variants. Pick one before you download anything. | Variant | File size | How sure are we? | | --- | --- | --- | | Ternary | 1.21 GB | Unconfirmed. From a [PrismML X post](https://x.com/PrismML/status/2059339163061940524), not yet checked against the model card. | | 1-bit | 0.93 GB | Unconfirmed. From a [PrismML X post](https://x.com/PrismML/status/2059339159899390326), not yet checked against the model card. | We have not yet captured the Hugging Face model card URLs for either variant. The demo README should link them. When you run the demo, note the size on disk of the file you actually downloaded, because that confirms or corrects the figures above. ## Set up and run the demo The README in the demo repo is the authority for install and run commands. Step 1 is known from the official repo URL. Steps 2 to 4 are inferred, so follow the README where it differs. ```bash # 1. Clone the official demo (source: repo URL above) git clone https://github.com/PrismML-Eng/Bonsai-image-demo cd Bonsai-image-demo # 2. INFERRED: install dependencies as the README describes # (note the exact command and the lockfile or commit you used) # 3. INFERRED: download the 1-bit or ternary model file # from the official model card linked in the README # (note the exact filename, its checksum, and its size on disk) # 4. INFERRED: run the demo and generate one image # (note the exact command, your prompt, and the settings) ``` You have succeeded when the demo writes an image file to disk. As a second check, disconnect from the network after the model download and generate another image. If that works, the demo runs fully offline. ## Record your run Fill in this block while you run the demo, then submit it as a device report. ```markdown - Date: - Operator: - Device: - OS: - CPU: - GPU: - RAM / VRAM: - Runtime (from demo README): - Demo repo commit: - Model variant: 1-bit / ternary - Model file + size on disk: - Prompt: - Image resolution / steps / settings: - Time to first image: - Total generation time: - Peak RAM / VRAM: - Offline mode verified: yes/no - Output image saved at: ``` Two of these numbers are the ones readers ask about first. Time to first image and total generation time are the image equivalents of time to first token and tokens per second in text generation. If the demo reports steps per second, record that too. Peak RAM and VRAM tell the next person whether their device can run the same variant. ## Troubleshoot | Symptom | Likely cause | Fix | | --- | --- | --- | | Install fails | The README changed or a dependency drifted. | Re-run from a pinned repo commit and file an issue upstream. Keep the commit hash and the error log. | | Model does not load | Wrong variant or file for the runtime the demo uses. | Match the runtime compatibility the model card states. Keep the exact filename and error. | | Generation is slow | The demo fell back to CPU instead of the GPU path. | Confirm the accelerator path in the README and note the hardware flags you set. | | Works online, fails offline | The runtime fetches an asset at generation time. | Find the fetched asset in your network log, download it ahead of time, and retest. | ## Next steps - [Record what you measured as a device report](https://yourwildcard.ai/docs/benchmarks/device-report-schema.md) so the next person can reproduce your run. - [Try Bonsai Image in the browser](https://yourwildcard.ai/docs/build-and-run/bonsai-image-webgpu.md) if you want the WebGPU path instead of a local install. - [Run the Bonsai text model with llama.cpp](https://yourwildcard.ai/docs/build-and-run/bonsai-llamacpp.md) to compare text and image generation on the same device. See something wrong? [Fix it](https://yourwildcard.ai/docs/contribute/contributor-guide.md). --- --- title: Bonsai Image In The Browser (WebGPU) status: published audience: DevRel engineer owner: Recipe author source_tier: primary_links benchmark_status: not_run last_reviewed: 2026-07-07 canonical_url: https://yourwildcard.ai/docs/build-and-run/bonsai-image-webgpu --- # Bonsai Image in the browser Bonsai Image 4B is PrismML's small image generation model, and it can run entirely inside a WebGPU browser with no server. On this page you can load the official demo, generate one image on your own machine, and record how it performed. The steps below come from the official announcement and demo repo, but we have not yet run them on a machine. Where a step is a guess, we say so. Do not cite numbers from this page in talks or posts until someone records a run. ## Check that your browser has WebGPU You need a browser with WebGPU enabled and a GPU that the browser can see. Open the browser's GPU status page, e.g., `chrome://gpu` in Chrome, and check that WebGPU is not disabled. This check is general practice, not a PrismML instruction. The exact browsers and minimum versions the demo supports are not confirmed yet, so check the [demo repo README](https://github.com/PrismML-Eng/Bonsai-image-demo) for the current list. The model comes in two variants. The release thread on X described a 1-bit variant at 0.93 GB and a ternary variant at 1.21 GB, both under an Apache 2.0 license. These three facts come from X posts and are unconfirmed until someone checks them against the model card and the repo LICENSE. The minimum VRAM the demo needs is not documented anywhere yet. If you are unsure, start with the smaller 1-bit variant. ## Run the demo 1. Open the official demo. Start from the [Bonsai Image 4B announcement](https://prismml.com/news/bonsai-image-4b), and if PrismML hosts a live demo, use it for your first run and note its URL. Otherwise use the [demo repo](https://github.com/PrismML-Eng/Bonsai-image-demo). 2. If the repo needs a local build, take the exact commands from its README. A typical path for a static WebGPU demo looks like this, but we have not confirmed it against the repo: ```bash # INFERRED, replace with the exact commands from the demo repo README git clone https://github.com/PrismML-Eng/Bonsai-image-demo cd Bonsai-image-demo npm install npm run dev ``` 3. Generate one image. Use the same prompt across runs so results are comparable between machines. 4. Reload the page with the network disabled and generate a second image. This tests whether the demo works offline once the model is cached. You have succeeded when an image appears in the browser and the network panel shows no inference request leaving your machine. ## Record what you measured Copy this block, fill it in during your run, and save it as a device report. ```markdown - Date: - Operator: - Device: - OS: - GPU (as reported by the browser, e.g. chrome://gpu): - RAM / VRAM: - Browser + exact version: - WebGPU status (enabled by default / flag required): - Demo commit or hosted demo URL: - Model variant (1-bit / ternary) and file size on disk or in cache: - Prompt: - Image output (resolution, saved file): - Model load time (cold): - Model load time (warm / cached): - Time to first image (TTFI): - Per-image latency after warmup: - Peak browser tab memory (task manager): - Offline behavior after first load: yes/no ``` The load times matter most, because on the browser path most of the wait is the model download. Also note the file size you actually downloaded, so we can confirm or correct the 0.93 GB and 1.21 GB figures from the release thread. ## Fix common problems If WebGPU is unavailable, your browser may be too old, your GPU may be blocklisted, or a flag may be required. Update the browser, check the GPU status page, and note any flag you had to set. | Symptom | Likely cause | Fix | | --- | --- | --- | | Model download stalls or fails | Large file over a slow network, or a host issue | Retry, and record the model URL and file size from the network panel | | Tab crashes during generation | Not enough GPU memory for the chosen variant | Switch to the smaller 1-bit variant and note your VRAM | | Slow generation on capable hardware | The browser fell back to a software path | Check which adapter the browser selected on the GPU status page | ## Next steps - [Device report schema](https://yourwildcard.ai/docs/benchmarks/device-report-schema.md). Turn your filled-in record into a device report others can compare against. - [Bonsai with llama.cpp](https://yourwildcard.ai/docs/build-and-run/bonsai-llamacpp.md). The reference recipe for running the text model locally. - [Runtime map](https://yourwildcard.ai/docs/build-and-run/runtime-map.md). See every other way to run the Bonsai family. See something wrong? [Fix it](https://yourwildcard.ai/docs/contribute/contributor-guide.md). --- --- title: Bonsai Studio on iPhone status: published audience: DevRel engineer owner: Recipe author source_tier: discovery benchmark_status: not_run last_reviewed: 2026-07-07 canonical_url: https://yourwildcard.ai/docs/build-and-run/bonsai-studio-ios --- # Bonsai Studio on iPhone Bonsai Studio is PrismML's iPhone app for Bonsai Image 4B, a small open-weight image model that generates images on the phone itself. On this page you can install the app, generate one image fully on-device, and measure how long it takes. The model itself is well documented. PrismML's official [release post](https://prismml.com/news/bonsai-image-4b) confirms that Bonsai Image 4B exists, ships in 1-bit and ternary variants, and is licensed Apache 2.0, and the [demo repo](https://github.com/PrismML-Eng/Bonsai-image-demo) shows local image generation working in code. The iPhone app is the unverified part. We have not run these steps on hardware yet, so treat each step as a plan to verify, not a confirmed procedure. The only source that describes the Bonsai Studio app is one X post, [PrismML status 2059339168250253731](https://x.com/PrismML/status/2059339168250253731), and X posts are unconfirmed. We have not yet found the App Store link, the supported device list, the minimum iOS version, or the download size in official PrismML or Apple sources. The steps below tell you how to capture them as you go. ## Start from the official sources - [Bonsai Image 4B release post](https://prismml.com/news/bonsai-image-4b). This is the model announcement. Check here first for an official app link. - [Bonsai Image demo repo](https://github.com/PrismML-Eng/Bonsai-image-demo). This is the reference implementation of local image generation. - [PrismML home](https://prismml.com/). Scan it for a Bonsai Studio product or app page. - [Bonsai Studio launch post on X](https://x.com/PrismML/status/2059339168250253731). It says PrismML launched Bonsai Studio for trying Bonsai Image 4B on iPhone with on-device generation. This claim is unconfirmed until an official page says the same. ## Check what you need You need an iPhone with about 2 GB of free storage. The model itself is small. PrismML describes the ternary variant as 1.21 GB and the 1-bit variant as 0.93 GB, though those numbers come from X posts ([ternary](https://x.com/PrismML/status/2059339163061940524), [1-bit](https://x.com/PrismML/status/2059339159899390326)) and you should confirm them against the release post or model cards before you cite them. The 2 GB figure is our own estimate to cover the model plus the app, not an official requirement. | Requirement | What we know | Source | | --- | --- | --- | | App | Bonsai Studio on iOS. No App Store URL captured yet. | [X post](https://x.com/PrismML/status/2059339168250253731), unconfirmed | | Device and OS | iPhone. Supported models and minimum iOS version unknown. | [X post](https://x.com/PrismML/status/2059339168250253731), unconfirmed | | Model | Bonsai Image 4B, 1-bit and ternary variants, Apache 2.0. Whether the app bundles or downloads it is unknown. | [Release post](https://prismml.com/news/bonsai-image-4b), official | | Free storage | About 2 GB suggested | Our estimate from the size claims above | ## Generate an image on your iPhone You can do the first three steps from a desk without an iPhone. Steps 4 to 8 need the phone. 1. **Find the official app link.** Check [prismml.com](https://prismml.com/), the [Bonsai Image 4B post](https://prismml.com/news/bonsai-image-4b), and the [@PrismML](https://x.com/PrismML) profile link for an App Store URL. Record the exact URL and the developer name shown on the listing, so you know the app really comes from PrismML. 2. **Capture the App Store details.** From the listing, record the minimum iOS version, download size, supported devices and chips, in-app purchases, and what the privacy label says about data collection. These replace the unknowns in the table above. 3. **Cross-check the model claims.** Confirm the Apache 2.0 license and the 1.21 GB and 0.93 GB file sizes against the [release post](https://prismml.com/news/bonsai-image-4b) and the Hugging Face model cards, not the X thread. 4. **Install and launch.** Install the app on your iPhone and note the model name and chip. Record whether the model weights download inside the app, with size and duration on Wi-Fi, or ship inside the app binary. 5. **Generate one image online.** Use this fixed prompt so runs are comparable across devices: ```text A bonsai tree on a desk beside a laptop, soft morning light, photorealistic. ``` Record which model variant, 1-bit or ternary, the app used if it shows you the choice. 6. **Verify offline behavior.** Turn on Airplane Mode with Wi-Fi off, force-quit and relaunch the app, and generate a second image with the same prompt. Record whether it works and whether any features degrade. We inferred this check ourselves, because no official page documents the app's offline behavior yet. 7. **Measure.** Record time to first preview and total generation time with a stopwatch or a screen recording. Note how hot the phone gets and how much battery the run uses. Record peak memory if you can get it through Xcode Instruments over USB, and otherwise write "not measured". 8. **Record your run** with the template below. ## Record your run Fill in this block so someone else can reproduce your run on their own device. The [device report schema](https://yourwildcard.ai/docs/benchmarks/device-report-schema.md) explains each field. ```markdown - Date: - Operator: - App Store URL / developer name: - App version: - Device (iPhone model, chip): - iOS version: - Model variant (1-bit / ternary) and file size: - Model delivery (bundled / in-app download, size, duration): - Prompt: - Time to first preview: - Total generation time: - Peak memory (or "not measured"): - Free storage before/after: - Offline generation verified: yes/no (Airplane Mode, relaunch) - Image output: attached / described ``` ## Fix common problems Nobody has tested these fixes yet. They are the failure modes we expect, and the notes tell you what to record if you hit one. | Symptom | Likely cause | What to record | | --- | --- | --- | | App not found on the App Store | Regional availability, TestFlight only, or a renamed listing | Search terms, region, and date. Check the PrismML site and X profile for the official link. | | Install blocked or crash on launch | Device or OS below the app's minimum | Exact iPhone model, chip, iOS version, and the listing's stated minimum | | Model download stalls or fails | Large download over cellular, or storage running out | Network type, free storage, and the download size shown in the app | | Generation fails offline | The app needs a network call despite the on-device claims | A screen recording of the Airplane Mode attempt and any error text | | Very slow generation or throttling | Older chip, low memory, or background load | Device model, battery level, ambient conditions, and back-to-back run times | ## Next steps - [Record what you measured](https://yourwildcard.ai/docs/benchmarks/device-report-schema.md) in the shared device report format. - [Run Bonsai Image on a laptop](https://yourwildcard.ai/docs/build-and-run/bonsai-image.md) to compare the same model off the phone. See something wrong? [Fix it](https://yourwildcard.ai/docs/contribute/contributor-guide.md). --- --- title: Ask The Docs Chat Agent status: verified audience: DevRel owner: Runtime validator source_tier: primary_links benchmark_status: partial last_reviewed: 2026-07-07 canonical_url: https://yourwildcard.ai/docs/build-and-run/docs-chat --- # Ask the docs chat agent The docs chat agent is 1-bit Bonsai 1.7B answering questions about this documentation, running entirely in your browser over WebGPU. On this page you learn how to use it, what quality bar it is held to, and where it will fail you. Open it at [Ask the docs](/chat). The first use downloads the 290 MB 1-bit model once from [onnx-community/Bonsai-1.7B-ONNX](https://huggingface.co/onnx-community/Bonsai-1.7B-ONNX) (Apache 2.0); after that it loads from cache and works offline. Nothing you type leaves your device, because there is no server to send it to. ## Know how it answers The agent does not browse. Before each answer it searches [/llms-full.txt](https://yourwildcard.ai/llms-full.txt), the same machine-readable bundle coding agents use, picks the two most relevant pages, extracts the paragraphs that match your question, and instructs the model to answer only from those excerpts and to say "The docs do not cover that yet" otherwise. Every answer lists its source pages as links. ## Watch the inference trace While the model works, the chat shows each stage of the inference chain as it happens: retrieve, assemble, prefill, and decode, each labeled with the design decision behind it and the number your own machine just measured, e.g. prompt tokens, time to first token, and tokens per second. After the answer, "How this answer was made" reopens the trace, ending with the whitepaper's intelligence density benchmark (2.832 per GB for 1-bit Bonsai 1.7B, the highest in its comparison). Most chat products hide this surface. We label it instead, because the moment you are watching a model think is the moment you are most curious about why a 290 MB model can think at all. The stages map one-to-one to the [technical guides](https://yourwildcard.ai/docs/technical-guides/kv-cache.md), so the trace doubles as a table of contents for the deep track. ## Hold it to a support bar Support products in this category, e.g. Fin by Intercom, publish resolution rates and hold their bots to two rules: never invent facts, and hand off honestly when unsure. We apply the same two rules, in this order of importance: 1. **No invented numbers.** An answer that fabricates a file size or benchmark score is worse than no answer. 2. **Grounded resolution.** The answer resolves the question using only what the docs say, with sources linked. The current gold-question results, run through the same prompt template on Bonsai 8B locally (llama.cpp, M4 Pro, greedy, seed fixed): | Question | Result | | --- | --- | | How big is the Bonsai 8B GGUF file? | Grounded: found 1.15 GB, flagged its verification status | | What license are the Bonsai models under? | Grounded: Apache 2.0, cited the model card evidence | | Energy per token on M4 Pro with MLX? | Grounded after a prompt fix: 0.074 mWh/token (the first prompt ordering missed it and abstained) | | Does compressing weights shrink the KV cache? | Grounded: correct no, with the right mechanism | Across all runs the model never invented a number; its failure mode was abstaining when the answer was present, which is the failure mode you want. The in-browser model is the smaller 1.7B, so expect more abstaining than the 8B shows here. The eval prompts and method live in [the design story](https://yourwildcard.ai/docs/learning-paths/designing-the-docs-chat.md). ## Expect these limits - A 1.7B model misreads dense tables more often than an 8B. Check the linked sources before quoting a number onward. - Retrieval is keyword-based over page text. Questions using words the docs do not use can miss the right page. - The first load needs a WebGPU browser (recent Chrome, Edge, or Safari) and a 290 MB download. - It answers about these docs only. It is not a general assistant. ## Next steps - [Try it](/chat) and compare the tokens per second your GPU reports against the [whitepaper's platform table](https://yourwildcard.ai/docs/prismml/whitepaper-benchmarks.md). - [Read how it was designed](https://yourwildcard.ai/docs/learning-paths/designing-the-docs-chat.md) with the two Hard Way courses. See something wrong? [Fix it](https://yourwildcard.ai/docs/contribute/contributor-guide.md). --- --- title: KV Cache status: published audience: DevRel engineer owner: Technical writer source_tier: course_material benchmark_status: not_run last_reviewed: 2026-07-07 canonical_url: https://yourwildcard.ai/docs/technical-guides/kv-cache --- # KV cache The KV cache is the block of memory where a language model stores the keys and values it has already computed for past tokens, so it does not recompute them for every new token. On this page you will learn why the cache exists, how to estimate its size for a given model and context length, and why a small weight file can still use gigabytes of memory at long context. ## Understand why the cache exists Attention requires every new token to look at the keys and values of all previous tokens. Without a cache, the model would recompute those keys and values for the whole growing sequence at every step. The KV cache stores them once. Each new step computes keys and values only for the newest token and reads the past from memory. This is why generation stays fast as the sequence grows, and it is also why memory use grows with every token you generate. ## Estimate the cache size Cache memory per token depends on the model's architecture. The formula multiplies five numbers: 2 (one copy for keys, one for values), the number of layers, the number of KV heads, the head dimension, and the bytes per element. Here is a worked example with the constants for a Qwen3-8B-class model (36 layers, 8 KV heads, head dimension 128), which match the published [Qwen3-8B model card](https://huggingface.co/Qwen/Qwen3-8B). Bonsai 8B is based on Qwen3-8B (see the [claim/source matrix](https://yourwildcard.ai/docs/prismml/claim-source-matrix.md)), so these constants apply to it. These numbers come from the course [Inference The Hard Way](https://chaiwithjai.github.io/inference-the-hard-way/); we have not measured them on hardware for this site. ```text bytes/token = 2 (K and V) x layers x kv_heads x head_dim x bytes_per_element = 2 x 36 x 8 x 128 x 2 = 147,456 bytes = about 144 KB per token ``` At 65,536 tokens: ```text 65,536 x 144 KB = about 9.2 GB ``` The exact number changes with the architecture. The point to remember is that cache memory scales with context length, layers, KV heads, head dimension, and element width. ## Explain why compression does not shrink the cache Quantizing or compressing a model makes the weight file smaller, but the KV cache is separate memory that the runtime allocates while it generates. If compression makes the weights much smaller, the cache can become the next memory limit. When a builder asks why a 4 GB model file still uses 12 GB of memory at long context, the answer is usually the cache. You can use the formula above to show them the arithmetic for their model. If you need a short version to say out loud: ```text The KV cache stores each token's keys and values so attention never recomputes the past. It grows with context length. Compressed weights can make the model file smaller, but long context can still be dominated by cache memory. ``` ## Go deeper - [Inference The Hard Way](https://chaiwithjai.github.io/inference-the-hard-way/) covers this arithmetic in full and is the course this page draws from. - The [vLLM docs](https://docs.vllm.ai/) describe paged KV management, which is how production servers keep the cache from fragmenting memory. ## Next steps - Read [Prefill vs decode](https://yourwildcard.ai/docs/technical-guides/prefill-vs-decode.md) to see how the cache is built during prefill and read during decode. - If you want to see cache growth on your own machine, run [Bonsai in llama.cpp](https://yourwildcard.ai/docs/build-and-run/bonsai-llamacpp.md) and watch the reported cache allocation at two context lengths. See something wrong? [Fix it](https://yourwildcard.ai/docs/contribute/contributor-guide.md). --- --- title: Prefill Versus Decode status: published audience: DevRel engineer owner: Technical writer source_tier: course_material benchmark_status: not_run last_reviewed: 2026-07-07 canonical_url: https://yourwildcard.ai/docs/technical-guides/prefill-vs-decode --- # Prefill vs decode Prefill and decode are the two phases of running a language model. Prefill reads your whole prompt at once, and decode generates the answer one token at a time. On this page you learn why the two phases hit different hardware limits, and how to measure each one so your speed claims are precise. This guide pairs with Lab 04 of [Inference The Hard Way](https://yourwildcard.ai/docs/learning-paths/inference-the-hard-way.md). ## Tell the two phases apart When you send a prompt to a runtime like llama.cpp, the runtime first processes every prompt token in one batch. This is prefill. Because the work is batched, the chip's compute units are usually the limit, and the number you feel is time to first token. After prefill, the runtime generates the reply one token at a time. This is decode. For each new token, the runtime streams the model weights from memory again, so memory bandwidth is usually the limit, and the number you feel is the gap between tokens. | Phase | What happens | Usual bottleneck | What to measure | | --- | --- | --- | --- | | Prefill | The runtime processes prompt tokens in parallel. | Compute, because work is batched across prompt tokens. | Prompt processing throughput and time to first token. | | Decode | The runtime generates one token at a time. | Memory bandwidth, because weights are streamed repeatedly. | Generation throughput and the time between tokens. | ## Understand why compression helps decode more During decode with a batch size of one, you can approximate the top speed of a generation step with: ```text max tokens/sec ~= memory bandwidth / bytes per forward pass ``` If a model format moves fewer bytes per forward pass, decode can get much faster. Prefill may not improve by the same ratio, because prompt tokens already share weight reads across parallel work. So a claim like "compression made the model 3x faster" is incomplete until it says which phase it measured. ## Measure both phases llama.cpp ships a benchmark tool that measures the two phases separately. Run it like this: ```bash ./build/bin/llama-bench -m -p 512 -n 128 ``` The `-p 512` flag measures prefill on a 512-token prompt, reported as `pp512` in tokens per second. The `-n 128` flag measures decode over 128 generated tokens, reported as `tg128`. Run the same command on a baseline model, e.g., an FP16 or Q4 build of the same weights, so you can compare like with like. This command comes from the upstream llama.cpp docs; we have not yet recorded a run on hardware. ## Write a precise speed claim Never write "Bonsai is X times faster" without saying: - Which phase, prefill or decode. - Which model and baseline. - Which runtime. - Which device. - Which benchmark command. - Which date and commit or release. A claim scoped this way holds up when someone else reruns it. A claim without this scope cannot be checked, so treat it as unverified. ## Next steps - [Weights on disk](https://yourwildcard.ai/docs/technical-guides/weights-on-disk.md) explains the model formats whose byte counts set the decode speed limit. - [KV cache](https://yourwildcard.ai/docs/technical-guides/kv-cache.md) covers the memory that grows during decode, if you have not read it yet. See something wrong? [Fix it](https://yourwildcard.ai/docs/contribute/contributor-guide.md). --- --- title: Weights On Disk status: published audience: DevRel engineer owner: Technical writer source_tier: course_material benchmark_status: inspectable last_reviewed: 2026-07-07 canonical_url: https://yourwildcard.ai/docs/technical-guides/weights-on-disk --- # Weights on disk Quantization means storing a model's weights with fewer bits per weight than the precision they were trained in, so the file on disk gets smaller. A "1-bit model" takes this to the extreme and stores each weight as a single sign bit plus a shared scale for its group, so a weight that would take 16 bits in FP16 takes about 1.125 bits on disk. On this page you learn how those bits are laid out in a block and how to decode one block back into weights with a few lines of Python. GGUF is the file format that llama.cpp and related runtimes use to store a model as one file, holding the quantized weights plus the metadata a runtime needs to load them. The Bonsai files you download from Hugging Face, e.g., `Bonsai-8B-Q1_0.gguf`, are GGUF files. A ternary model is a close relative of a 1-bit model. Each weight takes one of three values, -1, 0, or +1, times the group scale, instead of two. The extra zero value costs a little compression and buys back some quality. This guide pairs with Lab 05 of [Inference The Hard Way](https://yourwildcard.ai/docs/learning-paths/inference-the-hard-way.md), which has you parse one weight tensor by hand. Once you can do that, you can open a compressed model file and confirm byte by byte what it holds. ## Read the Q1 block layout The course describes a group of 128 weights stored as one FP16 scale plus 128 packed sign bits. To reconstruct a weight, you take its sign bit b and compute scale times (2b - 1), which gives plus or minus the scale. ```text group: scale: FP16 bits: 128 sign bits reconstruction: w = scale x (2b - 1) ``` The storage cost per weight follows from the block: ```text (2 bytes x 8 bits/byte + 128 bits) / 128 weights = 1.125 bits/weight ``` So "1-bit" is honest about the sign bits but rounds away the scale. The scale costs 16 bits per 128 weights, which adds the extra 0.125 bits. ## Decode a block in Python This sketch reads one block from an open file handle. It reads the 2-byte scale, unpacks the 16 bytes of sign bits, and reconstructs the 128 weights. ```python import numpy as np scale = np.frombuffer(f.read(2), dtype=np.float16)[0] bits = np.unpackbits(np.frombuffer(f.read(16), dtype=np.uint8)) weights = scale * (2.0 * bits - 1.0) ``` This sketch comes from the course material, and we have not yet run it against a real model file. Before you trust it on a real GGUF file, check the scale order, the bit order, the tensor offset, and the block struct against the runtime implementation and the GGUF metadata. ## Know what the file format does not give you A smaller file is not the whole story. The savings only help at inference time if the runtime keeps the weights packed during execution. The kernel must unpack the sign bits and multiply by the scale on the fly, without expanding the tensor back into a full FP16 copy in memory. If it expands the tensor first, you paid for compression on disk and got none of it at run time. ## Next steps - [Read the kernel](https://yourwildcard.ai/docs/technical-guides/kernel-reading.md) to see how the runtime unpacks these blocks during a matrix multiply. - [Build the bandwidth ledger](https://yourwildcard.ai/docs/technical-guides/bandwidth-ledger.md) to see why bits per weight sets the speed limit for decode. See something wrong? [Fix it](https://yourwildcard.ai/docs/contribute/contributor-guide.md). --- --- title: Kernel Reading status: published audience: DevRel engineer owner: Runtime educator source_tier: course_material benchmark_status: inspectable last_reviewed: 2026-07-07 canonical_url: https://yourwildcard.ai/docs/technical-guides/kernel-reading --- # Kernel reading Kernel reading means opening the runtime source code and tracing what a compute kernel actually does with a model's weights. On this page you will learn how to read a quantized matmul kernel and use what you find to confirm or reject a claim about a compressed model format. This guide pairs with Lab 06 of [Inference The Hard Way](https://yourwildcard.ai/docs/learning-paths/inference-the-hard-way.md). ## Know why the file size is not enough For a compressed format, file size alone does not settle a claim. A runtime could store small weights on disk and still expand them into full FP16 tensors in memory before it multiplies. The question that settles the claim is whether the runtime unpacks compressed weights inside the matmul path without expanding the full tensor in memory. The only place that answer lives is the kernel code, so you read the kernel. ## Answer five questions about the kernel Work through the kernel source and answer these five questions. Together they trace a weight from its packed form on disk to the multiply that uses it. 1. Which rows, columns, or groups does one thread own? 2. Where is the group scale loaded? 3. Where are packed sign bits loaded? 4. Which shifts, masks, or vector operations unpack the bits? 5. Where does multiply-accumulate happen, and in what precision? For example, in a llama.cpp quantized matmul kernel you would expect to find a loop where each thread loads a group scale, loads a block of packed bits, unpacks them with shift and mask operations, and accumulates the products into a float. If instead you find a step that writes a full FP16 copy of the tensor before the multiply, the format's memory claim does not hold in that backend. Once you can answer the five questions for one kernel, compare it against its neighbors. Each family below gives you a different reference point. | Kernel family | What to compare | | --- | --- | | Q1_0_g128 | Scale plus sign-bit unpack path. | | Q4_0 or Q8_0 | More conventional quantized unpack path. | | FP16 baseline | No compressed unpack path, and a different bandwidth profile. | | MLX or Metal | Backend-specific shader or array operation model. | | CUDA | Backend-specific thread/block strategy. | ## Write up what you found Record your reading as an annotated walkthrough that another engineer can review in five minutes. You do not need to write a new kernel. Fill in this template with line references from the source: ```text file: commit: backend: operation: compressed format: where scale loads: where bits load: where unpack happens: where accumulation happens: does it materialize FP16 weights in memory: open questions: ``` A finished walkthrough names the file, commit, and backend, answers the five questions with line references, and gives a clear yes or no on whether the kernel materializes FP16 weights in memory. Do not claim an implementation property from the whitepaper alone when the code is available. Read the runtime source, or label the claim as coming from the paper only. ## Next steps - [Build a bandwidth ledger](https://yourwildcard.ai/docs/technical-guides/bandwidth-ledger.md) to turn what the kernel loads into a bytes-per-token estimate. - [Review weights on disk](https://yourwildcard.ai/docs/technical-guides/weights-on-disk.md) if you need a refresher on how the packed formats are laid out before the kernel reads them. See something wrong? [Fix it](https://yourwildcard.ai/docs/contribute/contributor-guide.md). --- --- title: Bandwidth Ledger status: published audience: Benchmark contributor owner: Benchmark owner source_tier: course_material benchmark_status: not_run last_reviewed: 2026-07-07 canonical_url: https://yourwildcard.ai/docs/technical-guides/bandwidth-ledger --- # Bandwidth ledger A bandwidth ledger is a record that puts a predicted decode speed next to a measured one, so you can see how close a device gets to its physical limit. On this page you predict the decode speed of a model on your device from two numbers, run a benchmark, and write both results down side by side. ## Predict the decode ceiling When a model generates one token at a time, the main cost is moving the model's bytes from memory to the compute units. That gives a simple ceiling on decode speed: ```text max tok/s ~= memory_bandwidth / bytes_per_forward_pass ``` For example, if your device can move 100 GB per second and the model file is 5 GB, the ceiling is about 20 tokens per second. A measured rate above that number means you measured wrong. A measured rate far below it means the runtime is leaving speed on the table. Make the prediction twice, once for the full-precision model and once for the compressed one: ```text FP16 bytes/pass: record exact model size Q1_0 bytes/pass: record exact model size device bandwidth: record source and method ``` Write down where the bandwidth number came from, e.g., the vendor spec sheet or a memory benchmark you ran. The two sources can differ a lot. ## Fill your ledger row After the benchmark run, write one row per model with these values: - Bytes per forward pass, which is the exact model file size for batch-1 decode. - Predicted tokens per second from the formula above. - Measured tokens per second from your benchmark run. - Utilization, which is measured divided by predicted. This formula and template come from the Inference The Hard Way course. We have not yet recorded a measured run on hardware, so there is no filled example row to compare against yet. ## Compute energy per token To compute energy per token, you need both power and token rate: ```text energy per token = power / (3.6 x tokens_per_second) ``` If power rises but tokens finish much faster, energy per token can still fall. Do not publish an energy claim without the device, the measurement method, the token rate, and the formula. ## Report what you measured A report that someone else can check includes: - Device model and the source of its memory bandwidth number. - Runtime name and commit. - Model file and format. - The benchmark command you ran. - Prompt processing throughput. - Decode throughput. - Peak memory. - Power or energy measurement, if you have one. - An interpretation that is scoped to the phase and hardware you measured. ## Next steps - Read [Architectures](https://yourwildcard.ai/docs/technical-guides/architectures.md) to see how model design changes the bytes you just counted. - Take the [Inference The Hard Way](https://chaiwithjai.github.io/inference-the-hard-way) course to measure each of these numbers yourself, lab by lab. See something wrong? [Fix it](https://yourwildcard.ai/docs/contribute/contributor-guide.md). --- --- title: Architectures Beyond The Transformer status: published audience: DevRel engineer owner: Docs librarian source_tier: primary_links benchmark_status: not_run last_reviewed: 2026-07-07 canonical_url: https://yourwildcard.ai/docs/technical-guides/architectures --- # Architectures beyond transformers State space models and hybrid designs are alternatives to the transformer that trade some recall ability for much lower compute and memory at long context. On this page you can learn why these alternatives exist, how Mamba and Liquid's LFM2 approach the problem, and when you would pick one over a standard transformer for a device with limited memory. ## Understand why transformers are the default The transformer ([Vaswani et al., 2017](https://arxiv.org/abs/1706.03762)) replaced recurrence with self-attention, where every token can attend to every other token in the context. That design has three properties the current ecosystem is built on. 1. Training is fully parallel across sequence positions, which is what made large-scale pretraining affordable. 2. The model has direct random access to any earlier token, which favors in-context learning, retrieval-style prompting, and precise copying. 3. A decade of tooling assumes attention. Kernels (FlashAttention), serving stacks (vLLM, SGLang), file formats, and fine-tuning stacks are all built around it. The cost shows up at inference. Attention compute at prefill grows with the square of the sequence length. Decoding needs a KV cache that grows with context length, and that cache dominates memory at long context or high batch. Every architecture on this page is a response to that cost profile. ## Learn what state space models change [S4](https://arxiv.org/abs/2111.00396) showed that structured state space models could be trained efficiently as sequence models and handle very long-range dependencies. Its main result was strong performance on the Long Range Arena suite, including the Path-X task where prior sequence models had failed ([S4 repo](https://github.com/state-spaces/s4)). [Mamba](https://arxiv.org/abs/2312.00752) made SSMs competitive as language model backbones by adding selectivity. The state transition parameters depend on the input, so the model can decide what to keep in its fixed-size state. The paper and [repo](https://github.com/state-spaces/mamba) report these properties: - Compute grows linearly with sequence length, with no quadratic attention step. - Decoding is recurrent with a fixed-size state, so there is no KV cache that grows with context. - The paper reports roughly 5x higher inference throughput than a comparable transformer, and quality matching transformers of about twice the size at the scales tested (up to about 3B parameters). For example, if you run a chat session with a 100k-token history on a laptop, a transformer must hold a KV cache for all 100k tokens in memory, while a Mamba-style model holds only its fixed-size state. ## Decide between an SSM and a transformer An SSM is strongest when your constraint is memory and throughput on long streams at small batch, which is the typical edge and local profile. A transformer wins when the task needs precise retrieval from context, or when you need the mature serving stack today. The table below summarizes the tradeoffs. | Dimension | SSM (Mamba-style) | Transformer | Source | | --- | --- | --- | --- | | Long-context cost | Linear compute, constant-size state | Quadratic prefill, KV cache that grows linearly | [Mamba](https://arxiv.org/abs/2312.00752), [Attention](https://arxiv.org/abs/1706.03762) | | Decode memory at batch 1 | Fixed state, favorable for edge devices | KV cache scales with context length | [Mamba](https://arxiv.org/abs/2312.00752) | | Exact recall and copying from context | Weaker, because the state is a lossy compression of history | Strong, with direct attention to any token | [Mamba paper, discussion](https://arxiv.org/abs/2312.00752) | | In-context learning, few-shot prompting | Less proven at frontier scale | Ecosystem default, best documented | [Attention](https://arxiv.org/abs/1706.03762) | | Kernel and runtime support | Custom scan kernels and thinner runtime coverage | FlashAttention, vLLM, SGLang, and llama.cpp all assume attention | [Mamba repo](https://github.com/state-spaces/mamba) | | Frontier-scale evidence | Validated to low billions of parameters in the paper | Validated at every deployed scale | [Mamba](https://arxiv.org/abs/2312.00752) | ## See how hybrids split the difference Liquid AI's LFM2 family is the clearest public example of designing the architecture for the deployment target rather than compressing a transformer afterward. Per the [LFM2 report](https://arxiv.org/html/2511.23404v1) and [Liquid docs](https://docs.liquid.ai/lfm/getting-started/welcome), LFM2 uses a hybrid backbone that mixes gated short-convolution blocks with a small number of attention blocks. Liquid selected the mix with hardware-in-the-loop search that targets prefill and decode latency on the device. The [model library](https://docs.liquid.ai/lfm/models/complete-library) lists the released checkpoints, sizes, and supported deployment paths. Check the licensing terms per checkpoint there before you recommend commercial use, because we have not confirmed them. The hybrid design keeps a few attention layers for recall and in-context ability, and replaces the rest with operators whose cost is linear. This resolves the same tradeoff the table above describes, per layer instead of per model. PrismML built the Bonsai models to address edge constraints with compression (1-bit and ternary weights on a transformer-style backbone), not with a new architecture. Architecture choice and compression compound, so a model can use both. ## Compare the three edge strategies There are three ways to fit a model onto a small device. Liquid changes the architecture. PrismML's Bonsai changes the weight format. A conventional small transformer is quantized after training with a format such as GGUF or AWQ. | Property | Liquid LFM2 | PrismML Bonsai 8B | Conventional small transformer | | --- | --- | --- | --- | | Approach | Hybrid conv + attention, hardware-aware search ([LFM2 report](https://arxiv.org/html/2511.23404v1)) | Transformer-style backbone with 1-bit and ternary weights ([Bonsai 8B](https://prismml.com/news/bonsai-8b), [Ternary Bonsai](https://prismml.com/news/ternary-bonsai)) | Standard attention stack ([Attention](https://arxiv.org/abs/1706.03762)) | | What changes | The architecture | The weight format | Quantization after training | | Official local path | [Liquid model library](https://docs.liquid.ai/lfm/models/complete-library) | [Bonsai 8B GGUF](https://huggingface.co/prism-ml/Bonsai-8B-gguf), [MLX 1-bit](https://huggingface.co/prism-ml/Bonsai-8B-mlx-1bit) | llama.cpp, Ollama, or LM Studio | We have not yet run these models side by side on a shared device, so we do not publish speed, memory, or quality numbers for them. If you want to measure them yourself, start with the [Bonsai llama.cpp recipe](https://yourwildcard.ai/docs/build-and-run/bonsai-llamacpp.md). Vinod Khosla's X posts describe Bonsai 8B as about 1.15 GB and "more energy efficient". We have not confirmed those numbers against a primary source. Use the [official Bonsai 8B post](https://prismml.com/news/bonsai-8b) or the model card for size and efficiency numbers instead. ## Next steps - [Training and alignment](https://yourwildcard.ai/docs/technical-guides/training-and-alignment.md) covers how these backbones are trained and tuned. - [KV cache](https://yourwildcard.ai/docs/technical-guides/kv-cache.md) explains the memory cost that SSMs and hybrids are designed to avoid. See something wrong? [Fix it](https://yourwildcard.ai/docs/contribute/contributor-guide.md). --- --- title: Feature Stores And Data Pipelines status: published audience: DevRel engineer owner: Concept guide author source_tier: primary_links benchmark_status: not_applicable last_reviewed: 2026-07-07 canonical_url: https://yourwildcard.ai/docs/technical-guides/feature-stores --- # Feature stores A feature store is a system that computes a data value once, stores it with a timestamp and a key, and serves that same value to both your training jobs and your live agent. On this page you can learn when you need one, pick a platform, and run a checklist that keeps your training data and your serving data in agreement. Small-model agents fail most often at the data layer, not the model layer. A small local model has less capacity than a frontier model to compensate for noisy context, so a mismatch between what the model trained on and what it sees at inference time hurts a small model more. ## Understand the online and offline split A feature store keeps two copies of the same values because two consumers read them at very different speeds. Your training job reads months of history from the offline store, e.g., a warehouse or a set of Parquet files. Your agent reads one row per request from the online store, a low-latency key-value lookup. For example, a support agent that personalizes on "tickets opened in the last 30 days" trains on the historical value of that count as it stood at each past ticket, and at serving time it looks up the current count by user ID. | Consumer | Store | Access pattern | Failure if skipped | | --- | --- | --- | --- | | Training and fine-tuning jobs | Offline store (warehouse, lake, Parquet) | Point-in-time joins over history | Leakage, where the model trains on future data | | Agent at inference time | Online store (key-value, low latency) | Lookup by entity key per request | Skew, where the agent sees values training never saw | For LLM agents, "features" are broader than table columns. Retrieved documents, user and session state, tool-call results, embeddings, and the context assembled into the prompt all count. The same rule applies to all of them. Whatever context the agent gets at inference time must be something you can rebuild, correct as of that moment in time, when you generate the training data. If your agent is stateless (prompt in, answer out, no per-entity lookups) a feature store only adds operational cost. You need one when the agent personalizes on entity state (user, device, account) that changes over time and must match between training and serving. ## Pick a platform If you run a local-first or self-hosted stack, start with Feast. If you are already on Databricks, its feature store records lineage for you. If you want a managed platform that runs your pipelines, look at Tecton. Ray Data is not a feature store at all, but it does the heavy data processing before and after one. | Platform | Model | Core abstractions | Fit for small-model agents | | --- | --- | --- | --- | | [Feast](https://docs.feast.dev/) | Open source, bring your own infra | Entities, feature views, offline/online stores, `materialize` | Default for local-first and self-hosted stacks | | [Tecton](https://docs.tecton.ai/docs/concepts/tecton-concepts) | Managed platform | Data sources, feature views, feature services | Teams that want pipelines as code with managed orchestration and serving | | [Databricks Feature Store](https://docs.databricks.com/aws/en/machine-learning/feature-store/) | Bundled with Databricks and Unity Catalog | Feature tables, online tables, automatic lineage to models | Teams already on Databricks, since the platform records lineage automatically | | [Ray Data](https://docs.ray.io/en/latest/data/data.html) | Not a feature store. It is a distributed data processing tool | Datasets, `map_batches`, streaming execution | Multimodal preprocessing and offline batch inference feeding any of the above | Limits, pricing, and exact API surfaces change often. Treat the linked docs as the authority and this table as a routing decision only. ## Use Ray Data around the store Ray Data covers the pipeline work upstream and downstream of a feature store rather than replacing it. Use it for: - Distributed preprocessing of multimodal inputs (images, audio, documents) into features or embeddings before they land in an offline store. - Offline batch inference, which means running a small model over a large dataset to generate labels, embeddings, or synthetic training data. With streaming execution, the working set never has to fit in memory. - Mixed CPU and GPU pipelines, where decode and transform steps run on CPU workers while model steps run on GPU workers. The source for all of this is the [Ray Data docs](https://docs.ray.io/en/latest/data/data.html). ## Check training and serving consistency Run this checklist before you ship an agent that personalizes on stored state: 1. **Single definition.** Every feature is defined once, in a feature view or feature table, and both the training export and online serving read that definition. Do not copy SQL between the notebook and the service. 2. **Point-in-time joins.** Build training sets with as-of joins against event timestamps, never with "latest value" joins. 3. **Timestamp discipline.** Give every feature row an event timestamp and, ideally, an ingested timestamp, so you can detect data that arrives late. 4. **Freshness budget.** State a maximum staleness for each online feature, and document what the agent does on a cache miss or a stale read. 5. **Same transformations.** Pin tokenization, normalization, embedding model version, and prompt template version identically for dataset generation and serving. 6. **Skew measurement.** Log served feature values and periodically diff their distribution against the training export for the same window. 7. **Replayability.** Given a request ID, you can reconstruct the exact feature values and context the agent saw. ## Hunt for leakage Leakage means the model trained on information that was not available at prediction time. Check for it like this: - Grep every training join for reads of mutable tables that are not point-in-time. Each one is a suspected leak. - Verify that label timestamps are strictly later than all feature timestamps used to predict them. - For agent traces used as training data, strip fields that were only knowable after the labeled outcome, e.g., resolution codes, final status, or human corrections. - Hold out by entity and by time, not only by random row split. Random splits let the model memorize individual entities when you use personalization features. - Re-run your eval with each suspicious feature removed. A large accuracy drop from one feature is a sign of leakage, not a win. Leaked features raise offline metrics while online behavior gets worse, and the damage is easy to miss. Do not trust any offline agent accuracy number that does not state its split strategy and its point-in-time policy. ## Document your features and datasets Write these four documents as you build. None of them needs a GPU, and everything downstream of the data layer depends on them. | Artifact | Minimum fields | Where it lives | | --- | --- | --- | | Feature definition doc | Name, entity, source, transformation, owner, freshness budget, online/offline availability | Next to the feature repo or registry | | Lineage note | Raw source, transformations, feature, and the models that consume it. On Databricks, capture the [automatic lineage view](https://docs.databricks.com/aws/en/machine-learning/feature-store/) | Per feature table | | Dataset card | Provenance, collection window, license, split strategy, known leakage risks, intended use, contact | Alongside the dataset artifact | | Leakage check log | Checks run (from the section above), date, findings, fixes | Linked from the dataset card | ## Try a minimal Feast loop These commands come from the [Feast docs](https://docs.feast.dev/). We have not yet run them on hardware, so treat them as a starting point rather than a verified recipe. ```bash pip install feast feast init agent_features cd agent_features/feature_repo feast apply # register entities and feature views feast materialize-incremental $(date -u +%Y-%m-%dT%H:%M:%S) # load online store ``` To confirm the loop works on your machine, define one entity and one feature view over a local Parquet file. Then build a training dataframe with `get_historical_features` and confirm the point-in-time join drops future rows. Finally, read the same feature online with `get_online_features` and confirm the value matches the latest offline row. We also do not yet have a measured answer for which online store backend (SQLite, Redis, or an embedded key-value store) works best for a fully local agent stack, or for Ray Data throughput on small quantized models. The platform docs today focus on tabular ML, so agent-specific guidance is thinner than the rest of this page. ## Next steps - [Evaluation concepts](https://yourwildcard.ai/docs/technical-guides/evaluation-concepts.md). Once your data layer is clean, learn how to measure whether the model actually improved. - [Evaluation harness](https://yourwildcard.ai/docs/benchmarks/evaluation-harness.md). Turn those concepts into runs you can repeat and compare. See something wrong? [Fix it](https://yourwildcard.ai/docs/contribute/contributor-guide.md). --- --- title: Training, Fine-Tuning, And Preference Alignment status: published audience: DevRel engineer owner: Recipe author source_tier: primary_links benchmark_status: not_run last_reviewed: 2026-07-07 canonical_url: https://yourwildcard.ai/docs/technical-guides/training-and-alignment --- # Training and alignment Fine-tuning updates a pretrained model on your own data, and preference alignment then steers the model toward outputs people prefer. On this page you pick the right tool for an adaptation job, choose between LoRA and QLoRA, choose between DPO, PPO, and GRPO, format your data correctly, and check the tuned model for regressions before you ship it. ## Pick a training tool Five tools cover almost every adaptation job, and the choice comes down to your hardware and your method. For example, if you have one consumer GPU and want to fine-tune a small model with LoRA, start with [Unsloth](https://unsloth.ai/docs). If you instead need full RLHF across many machines, you need [OpenRLHF](https://github.com/OpenRLHF/OpenRLHF) on a [Ray](https://docs.ray.io/) cluster. | Situation | Use | Why | | --- | --- | --- | | Single GPU, small model, fast SFT or LoRA iteration | [Unsloth](https://unsloth.ai/docs) | Memory-efficient kernels tuned for single-device fine-tuning. | | Many methods (SFT, DPO, PPO, reward modeling) behind one config and CLI | [LLaMA-Factory](https://llamafactory.readthedocs.io/en/latest/) | You configure training in one YAML file across model families, without custom trainer code. | | Preference alignment inside the Hugging Face ecosystem | [TRL](https://huggingface.co/docs/trl/en/index) | Maintained trainers (SFTTrainer, DPOTrainer, PPOTrainer, GRPOTrainer) on top of Transformers. | | Full RLHF at multi-node scale with separate actor, critic, and reward roles | [OpenRLHF](https://github.com/OpenRLHF/OpenRLHF) | An RLHF framework scheduled on Ray and designed to distribute the four RLHF model roles. | | Multi-node orchestration, fault tolerance, or a cluster shared with data and serving workloads | [Ray Train](https://docs.ray.io/) | It uses the same Ray primitives as Ray Data and Ray Serve. | Do not introduce Ray Train for a job that fits on one machine. Ray Train helps when you need multi-node scaling, checkpoint and restore across worker failures, or a shared cluster with the data and serving layers. That is also why OpenRLHF builds on Ray ([OpenRLHF paper](https://arxiv.org/html/2405.11143v4)). ## Choose LoRA or QLoRA LoRA freezes the base model and trains small low-rank adapter matrices on top of it, so you update a tiny fraction of the parameters. QLoRA does the same but first quantizes the frozen base weights to 4-bit, which cuts VRAM further at a small quality cost. Per the [Unsloth fine-tuning guide](https://unsloth.ai/docs/get-started/fine-tuning-llms-guide), the practical decision is the precision of the frozen base weights versus your VRAM budget. For example, on a 24 GB consumer GPU you would pick QLoRA to fit the largest base model you can. | Method | Base weights | Trainable parameters | When | | --- | --- | --- | --- | | Full fine-tune | Updated, full precision | All | Rarely needed for small-model adaptation. Highest VRAM. | | LoRA | Frozen, 16-bit | Low-rank adapters only | Default for task adaptation. Adapters are easy to swap and ship. | | QLoRA | Frozen, quantized to 4-bit | Low-rank adapters only | Consumer GPUs, largest base that fits. Small quality tradeoff versus LoRA. | Per-model VRAM tables and supported model lists change often. Check the live matrix in the [Unsloth docs](https://unsloth.ai/docs) rather than trusting numbers copied into this page. ## Run a first fine-tune The shortest path to a working LoRA or QLoRA run is Unsloth plus TRL. These steps come from the [Unsloth fine-tuning guide](https://unsloth.ai/docs/get-started/fine-tuning-llms-guide). We have checked them against the upstream docs but have not yet run them on hardware, so treat them as a starting point rather than a proven recipe. ```bash pip install unsloth ``` ```python from unsloth import FastLanguageModel model, tokenizer = FastLanguageModel.from_pretrained( model_name="", max_seq_length=2048, load_in_4bit=True, # QLoRA path ) model = FastLanguageModel.get_peft_model(model, r=16, lora_alpha=16) # Then train with TRL's SFTTrainer on a chat-formatted dataset. ``` The equivalent LLaMA-Factory path is a YAML config plus `llamafactory-cli train .yaml`. See the [LLaMA-Factory docs](https://llamafactory.readthedocs.io/en/latest/) for the current config schema. We do not yet know the best-supported fine-tuning path for PrismML Bonsai models in Unsloth, and we found no primary source on whether 1-bit or ternary Bonsai checkpoints can be LoRA-tuned directly or must be adapted on a higher-precision parent first. Until a source or a recorded run settles those questions, fine-tune a standard 16-bit base model. ## Choose an alignment method DPO, PPO, and GRPO all steer a model toward preferred outputs, but they need different data and different amounts of hardware. Start with DPO when you have static preference pairs and want the simplest pipeline. Use GRPO when a program can score outputs, e.g., a math checker or a format validator. Use PPO, via [TRL](https://huggingface.co/docs/trl/en/index) or OpenRLHF at scale, only when you need an online loop with a learned reward model. | Method | Models in memory | Data required | Character | | --- | --- | --- | --- | | PPO | Policy, reference, reward, value/critic | Preference pairs to train the reward model, then prompts | Classic RLHF, with the most components and an online generation loop. | | DPO | Policy, reference | Preference pairs (chosen and rejected) | Turns RLHF into a single classification-style loss on preference data, with no RL loop ([DPO paper](https://arxiv.org/abs/2305.18290)). | | GRPO | Policy, reference (no value model) | Prompts plus a scoring function, sampled in groups of completions | Replaces the critic with group-relative baselines, and works well with verifiable rewards such as math and code checks. | We do not yet have an experiment log showing which GRPO reward functions work best for small local models, so the format-check versus task-grader choice is still open. ## Format your data Two formats recur across TRL, LLaMA-Factory, and Unsloth. SFT uses a conversational format, and DPO uses a preference format. ```json // SFT: conversational format {"messages": [ {"role": "system", "content": "..."}, {"role": "user", "content": "..."}, {"role": "assistant", "content": "..."} ]} ``` ```json // DPO: preference format {"prompt": "...", "chosen": "...", "rejected": "..."} ``` Follow these rules when you build a dataset: - Apply the model's own chat template before tokenizing. - Keep `chosen` and `rejected` as completions to the same `prompt`. - Never mix templated and raw-text rows in one dataset. The [TRL docs](https://huggingface.co/docs/trl/en/index) and the [LLaMA-Factory docs](https://llamafactory.readthedocs.io/en/latest/) data pages list the exact accepted schemas per trainer. ## Check for regressions before you ship Preference tuning optimizes for the preference signal, and it can weaken capabilities that were not in the preference set without an obvious sign. Before you promote a tuned model, do the following: 1. Freeze a regression prompt set before tuning, covering task prompts, formatting and instruction-following probes, refusal probes, and a few long-context items. 2. Run the base and tuned checkpoints on the identical set with identical sampling parameters, and diff the outputs side by side. 3. Track response length drift. Tuned models often win preferences by getting longer, not better. 4. Log KL divergence from the reference model where the trainer exposes it. TRL trainers do. 5. Record everything in the [device report schema](https://yourwildcard.ai/docs/benchmarks/device-report-schema.md) so someone else can reproduce the run. We have not yet published a reproducible alignment benchmark run, so do not cite quality deltas from this page. Cite a recorded run once one exists. ## Next steps - [Feature stores](https://yourwildcard.ai/docs/technical-guides/feature-stores.md) covers the data layer that feeds training jobs like these. - [Evaluation concepts](https://yourwildcard.ai/docs/technical-guides/evaluation-concepts.md) goes deeper on how to judge a tuned model. See something wrong? [Fix it](https://yourwildcard.ai/docs/contribute/contributor-guide.md). --- --- title: Evaluation For The Deployment Target status: published audience: Benchmark contributor owner: Recipe author source_tier: primary_links benchmark_status: not_run last_reviewed: 2026-07-07 canonical_url: https://yourwildcard.ai/docs/technical-guides/evaluation-concepts --- # Evaluation concepts Evaluation is how you check that a model still does its job on the device where it will actually run, not just on a cloud leaderboard. On this page you can pick the right evaluation style for your model, write a task card with an acceptance test, and set up a regression set that survives model and runtime upgrades. A local model must be judged on local tasks. That means you track device-level latency, memory, energy, privacy, and offline behavior alongside output quality. ## Choose an evaluation mode There are three common ways to evaluate a model, and each fits a different question. Arena-style evaluation asks real people which of two outputs they prefer. An automatic task suite scores the model against tasks you define, e.g., "summarize this document and pass an exact-match check". A regression set replays cases the model has passed before to catch anything a change broke. For example, if you quantize Bonsai 8B to a smaller file and want to know whether it is still usable, you would run your automatic task suite first because it is cheap, then check the regression set before you promote the new build. | Mode | What it measures | When to use it | Cost | | --- | --- | --- | --- | | Arena-style human preference | Which output real people prefer in pairwise, anonymized battles. | Ranking general chat quality and catching problems no static benchmark covers. | High, because it needs sustained voter traffic or a paid rater pool. | | Automatic task suite | Pass or fail on defined tasks with acceptance tests. | Domain-specific deployments and gating before you promote a model or quantization. | Low per run once the tasks are written. | | Regression set | Whether a new model, runtime, or quantization breaks cases that passed before. | Every upgrade, including a new model version, a new quant format, and a new runtime commit. | Low, but it needs maintenance discipline (see below). | ## Learn from the Arena lineage The human preference methods this ecosystem uses come from LMSYS and Berkeley. 1. **Chatbot Arena** ([launch blog](https://www.lmsys.org/blog/2023-05-03-arena/), [paper](https://arxiv.org/abs/2403.04132)) runs crowdsourced, pairwise, anonymized model battles scored with Elo-family rating systems. The paper details the Bradley-Terry model and the sampling method. The main finding is that live human preference on real prompts catches quality differences that static benchmarks miss. 2. **Arena-Hard-Auto** ([repo](https://github.com/lmarena/arena-hard-auto)) is an automatic pipeline that selects hard prompts from arena traffic and uses a language model as the judge for pairwise comparison. This removes human raters in exchange for repeatability and speed. Per the repo, it is designed to correlate with human arena rankings. Check the repo for the current correlation numbers before you cite them, because we have not verified them here. Three lessons transfer to local evaluation: - Pairwise scoring beats absolute scoring. - The prompt distribution should come from real usage. - An automated judge must be validated against human preference before you trust it for gating. ## Record more than quality Cloud arenas rank quality only. For local deployment, quality is one of six dimensions you should record. When you compare two local combinations of model, runtime, and quantization, record all of these: | Dimension | Metric | Why it gates deployment | | --- | --- | --- | | Quality | Pairwise win rate or task pass rate | The model must still do the job after compression. | | Latency | Time to first token, tokens per second | Interactive local use is not workable when the first token takes more than a few seconds. | | Memory | Peak RAM or VRAM, model file size | This determines which devices can run the model at all. | | Energy | Power draw or battery impact, when you can measure it | Edge and phone deployments are limited by energy. | | Privacy | What data leaves the device (it should be none) | Privacy is the main reason users choose local inference. | | Offline | Works with the network disabled after the model download | This separates truly local paths from paths that still need the cloud. | Record your hardware and software setup with the environment block from [the Bonsai llama.cpp recipe](https://yourwildcard.ai/docs/build-and-run/bonsai-llamacpp.md), run the comparison with [the evaluation harness protocol](https://yourwildcard.ai/docs/benchmarks/evaluation-harness.md), and report the results in [the device report format](https://yourwildcard.ai/docs/benchmarks/device-report-schema.md). ## Write a task card A task card describes one capability you plan to ship, e.g., local summarization on a phone, along with the bars it must clear. Write one task card per capability. Copy this template verbatim: ```markdown # Task: - Layer: evaluation and alignment - Deployment target: - User story: As a , I need so that . - Prompt set: - Scoring: - Judge: - Quality bar: = 85% pass rate on the prompt set> - Latency bar: = 10 tok/s on the target device> - Memory bar: - Offline required: yes/no - Owner: - Status: not_run | partial | reproducible | published ``` ## Write an acceptance test A task card is only useful when another person can run it and get a clear verdict. The acceptance test is the exact procedure they follow: ```markdown # Acceptance test: 1. Environment: record the device report block (device, OS, runtime commit, model file, quantization). 2. Run: . 3. Expected: . 4. Evidence: attach raw outputs, scores, TTFT, tok/s, peak memory. 5. Offline check: repeat one run with networking disabled; record result. 6. Verdict: pass/fail against every bar, not just quality. ``` These templates come from the upstream methods above. We have not yet run a full cycle from task card to verdict on hardware, so treat them as a starting point rather than a proven protocol. ## Maintain a regression set A regression set is the accumulated list of prompts and tasks a deployment must never fail again. Five rules keep it useful: 1. **Add on failure, not on success.** When a bug report or an arena loss gets fixed, add its prompt to the set. 2. **Pin everything.** Each entry records the model variant, quantization, runtime commit, device class, and the date it last passed. An entry without these pins is useless. 3. **Rerun on every supply-chain change.** A new model version, quant format, runtime commit, or kernel path can each change outputs without warning. 4. **Prune deliberately.** Retire an entry only with a written reason, e.g., the task is obsolete. Never retire one because it is inconvenient to keep passing. 5. **Separate quality regressions from performance regressions.** A model can keep its pass rate while its response time doubles. Both are failures against the deployment target. ## Next steps - Run these ideas against real numbers with [the evaluation harness](https://yourwildcard.ai/docs/benchmarks/evaluation-harness.md). - Record what you measure in [the device report format](https://yourwildcard.ai/docs/benchmarks/device-report-schema.md). See something wrong? [Fix it](https://yourwildcard.ai/docs/contribute/contributor-guide.md). --- --- title: Evaluation Harness status: published audience: Benchmark contributor owner: Benchmark owner source_tier: internal_standard benchmark_status: not_run last_reviewed: 2026-07-07 canonical_url: https://yourwildcard.ai/docs/benchmarks/evaluation-harness --- # Evaluation harness The evaluation harness is a fixed protocol for measuring how a model actually behaves on a real device. It covers answer quality, speed, memory use, and energy use when you can measure it. On this page you can pick the prompts to run, record your results in a standard report, and check that your report is complete enough for someone else to repeat. No one has run this protocol on hardware yet. Your run can be the first published report. ## Run the first task suite Run these prompts against your model and keep the raw output. The suite is small on purpose, so one session on one device is enough. - One simple explanation prompt. - One code or reasoning prompt. - One question-answering prompt over a local document. - One summarization prompt. - One long-context prompt, which puts pressure on the KV cache. - One image-generation prompt if you are testing a Bonsai Image path. This one is optional. If you have not set up a runtime yet, start with the [Bonsai on llama.cpp recipe](https://yourwildcard.ai/docs/build-and-run/bonsai-llamacpp.md) and come back here with a working model. ## Record your run Write down your results in a device report while the run is still on screen. The report is a plain list of fields. For example, under "Decode tok/s" you would write the tokens per second your runtime printed during generation, and under "Command" you would paste the exact command you ran, flags included. Copy this template and fill in every field you can: ```markdown - Date: - Operator: - Device: - OS: - CPU: - GPU: - RAM / VRAM: - Runtime: - Runtime commit: - Model: - Model file: - Quantization / format: - Command: - Prompt set: - Output sample: - TTFT: - Prompt processing tok/s: - Decode tok/s: - Peak memory: - Power / energy method: - Offline mode verified: - Failure notes: ``` Leave a field blank rather than guess. A blank field is honest and a guessed number is not. ## Know what each dimension needs Each measurement is only useful if you record enough context to repeat it. For example, a decode speed of 14 tokens per second means nothing without the model file, the quantization, and the device it ran on. | Dimension | What to record | | --- | --- | | Quality | The task, the prompt set or dataset, and how you scored the output. | | Latency | Time to first token, prompt processing speed, and decode speed. | | Memory | Peak RAM or VRAM, the model file, and the context length. | | Energy | The device, how you measured power, and the token rate at that power. | | Runtime | The runtime, its commit, the flags, and the backend path. | | Device fit | Whether it works offline, how hard install was, and any thermal notes. | ## Check that your report is complete A report is complete when someone else can repeat your run from the page alone. That means it names the model, the runtime, the device, the exact command, and the date, and it points to the raw output or a log. Do not cite a benchmark number anywhere on this site until its report meets that bar. ## Next steps - [Run Bonsai on llama.cpp](https://yourwildcard.ai/docs/build-and-run/bonsai-llamacpp.md) to produce your first numbers to report. - Submit your report row via the [contributor guide](https://yourwildcard.ai/docs/contribute/contributor-guide.md). See something wrong? [Fix it](https://yourwildcard.ai/docs/contribute/contributor-guide.md). --- --- title: Device Benchmark Schema And Task Suite status: published audience: Benchmark contributor owner: Benchmark owner source_tier: internal_standard benchmark_status: not_run last_reviewed: 2026-07-07 canonical_url: https://yourwildcard.ai/docs/benchmarks/device-report-schema --- # Device report schema A device report is a record of one benchmark run. It captures which model you ran, on which machine, with which runtime, and how fast it went, so that someone else can compare your numbers with theirs or repeat the run. On this page you can copy the report format, pick a standard task to measure, and learn how to mark what your row does and does not prove. ## Record a run Each report is one CSV row or one JSON object. Here is the CSV header to copy: ```csv date,operator,device,os,ram_gb,vram_gb,runtime,runtime_commit,model,model_file,quantization,prompt_set,ttft_ms,prompt_tok_s,decode_tok_s,peak_mem_gb,power_w,power_method,repro_notes,caveats,benchmark_status ``` Here is the same report as JSON. This example shows the shape only. It is not a measured result, which is why the metric fields are null. ```json { "date": "2026-07-06", "operator": "contributor-handle", "device": "MacBook Pro 14 M3 Pro", "os": "macOS 15.5", "ram_gb": 36, "vram_gb": null, "runtime": "llama.cpp", "runtime_commit": "", "model": "Bonsai-8B", "model_file": "Bonsai-8B-Q1_0.gguf", "quantization": "Q1_0 GGUF", "prompt_set": "suite-v1/summarization", "ttft_ms": null, "prompt_tok_s": null, "decode_tok_s": null, "peak_mem_gb": null, "power_w": null, "power_method": "", "repro_notes": "llama-cli command, flags, model card URL, log path", "caveats": "example row only", "benchmark_status": "not_run" } ``` If you want the step-by-step protocol for running a benchmark, see the [evaluation harness](https://yourwildcard.ai/docs/benchmarks/evaluation-harness.md). This page is the reference for the data that run produces. ## Fill in each field Every field is required unless the rule says otherwise. Field names are the CSV column names. | Field | Type | Rule | | --- | --- | --- | | `date` | ISO 8601 date | Date of the run, not of submission. | | `operator` | string | Your contributor handle. | | `device` | string | Marketed name plus chip, e.g. "MacBook Pro 14 M3 Pro". | | `os` | string | OS and version. | | `ram_gb` | number | Total system RAM. | | `vram_gb` | number or empty | Only when VRAM is separate from RAM. | | `runtime` | string | Runtime name, e.g. llama.cpp, vLLM, SGLang, MLX. | | `runtime_commit` | string | Commit hash or release tag. Without it no one can repeat your run. | | `model` | string | Model name as it appears on the model card. | | `model_file` | string | Exact filename, e.g. `Bonsai-8B-Q1_0.gguf`. | | `quantization` | string | Quantization or format label from the model card. | | `prompt_set` | string | Task suite ID from this page, e.g. `suite-v1/summarization`. | | `ttft_ms` | number | Time to first token, in milliseconds. | | `prompt_tok_s` | number | Prompt processing throughput. | | `decode_tok_s` | number | Decode throughput. This is the headline "tok/s". | | `peak_mem_gb` | number | Peak RAM or VRAM during the run. | | `power_w` | number or empty | Optional. Leave blank unless you name a measurement method. | | `power_method` | string or empty | Required whenever `power_w` is filled. | | `repro_notes` | string | Command or manual flow, flags, model card URL, log location. | | `caveats` | string | Thermal throttling, background load, first-run cache effects. | | `benchmark_status` | enum | `not_run`, `partial`, `reproducible`, or `published`. See below. | Leave a metric blank rather than estimating it. A blank field only means the row is incomplete. An invented number makes the whole row untrustworthy. ## Pick a task from the suite The task suite gives every run the same prompts, so a `decode_tok_s` number from your laptop can be compared with one from someone else's desktop. The suite ID is `suite-v1`, and each task ships as one prompt set. For example, `suite-v1/summarization` gives the model a fixed article and asks for a summary within a length bound. You record the speed numbers from that run, and you check that the summary kept the facts and stayed within the bound. | Task ID | What it exercises | Pass signal | | --- | --- | --- | | `suite-v1/coding` | One code generation or code reasoning prompt. | Output compiles or a human confirms correct logic. | | `suite-v1/summarization` | Fixed article to a summary within a length bound. | Facts are preserved and the length bound is respected. | | `suite-v1/extraction` | Structured fields from messy text into JSON. | Output parses and fields match a key. | | `suite-v1/tool-use` | Prompt requiring a correctly formed tool call. | Tool name and arguments are valid. | | `suite-v1/doc-qa` | Question over a local document in context. | Answer grounded in the document, not prior knowledge. | | `suite-v1/image-gen` | One image prompt, only for image-capable paths such as Bonsai Image. | Image renders. Size, steps, seed, and total latency are recorded. | Image runs also record the prompt, image size, steps, seed, and total latency. These fields match the image table in the device report template. ## Measure KV cache reuse The KV cache microbench measures how much faster a runtime answers when it can reuse work from an earlier prompt. Send N prompts that share one long fixed prefix and have short varying suffixes. Then compare time to first token between the first request, which is cold, and the later requests, which are warm. This protocol comes from the runtimes' own docs. No one has run it on hardware yet, so treat the steps as a starting point rather than a proven recipe. Run it on each runtime that serves your model, where supported: - [vLLM](https://github.com/vllm-project/vllm). Use the automatic prefix caching path. - [SGLang](https://github.com/sgl-project/sglang). Use the RadixAttention prefix reuse path. - [llama.cpp](https://github.com/ggml-org/llama.cpp). Use `llama-server` with prompt caching, as in the [Bonsai recipe](https://yourwildcard.ai/docs/build-and-run/bonsai-llamacpp.md). Record one report row per runtime. Set `prompt_set` to `kv-cache-v1` and put the warm and cold TTFT numbers in `repro_notes`. ## Check quality, not just speed A speed number is misleading if the model got faster by getting worse, so every complete report pairs its metrics with two kinds of quality evidence. 1. **Human rubric.** Record the task, the expected behavior, the observed behavior, your rating, any failure modes, and a path to the output. This matches the "Quality Notes" block of the device report template and stays separate from the latency and memory numbers. 2. **Automated checks.** Run parse checks for `extraction` and `tool-use` (valid JSON, valid tool call), length bounds for `summarization`, and exact-match or containment checks for `doc-qa`. A regression is any check that passed before and fails on a newer model file, quantization, or runtime commit with the same prompt set. When you find one, file it as its own row with `benchmark_status: partial` and name the failing check in `caveats`. Never overwrite the passing historical row. ## Mark what your row proves The `benchmark_status` field says how far a row can be trusted. Set it honestly. | Status | What it means | | --- | --- | | `not_run` | The protocol is written down and no one has run it. | | `partial` | You measured something real, but the row is incomplete. Any measurement with the environment fields filled qualifies. | | `reproducible` | Someone else can repeat your run from the row alone. That requires the command, runtime commit, model file, device, and log location, and a second person has repeated the run with results in an explained range. | | `published` | A maintainer has reviewed the row as a reference report. | ## Next steps - [Run Bonsai with llama.cpp](https://yourwildcard.ai/docs/build-and-run/bonsai-llamacpp.md) to produce your first row of data. - [Follow the evaluation harness](https://yourwildcard.ai/docs/benchmarks/evaluation-harness.md) for the step-by-step benchmark protocol. - Submit your report row via the [contributor guide](https://yourwildcard.ai/docs/contribute/contributor-guide.md). See something wrong? [Fix it](https://yourwildcard.ai/docs/contribute/contributor-guide.md). --- --- title: Source Policy status: published audience: Docs librarian owner: Source reviewer source_tier: internal_standard benchmark_status: not_applicable last_reviewed: 2026-07-07 canonical_url: https://yourwildcard.ai/docs/sources/source-policy --- # Source policy A source is any document, repository, post, or recording that a fact on this wiki comes from, and not every source is strong enough to support a published fact. On this page you can sort a source into one of five tiers and decide whether a claim from it is ready for a public page. ## Sort a source into a tier The tier tells you how much a source can support on its own. A runtime's official README is Tier 0, so you can cite it directly for a build command. An X post that claims a token rate on a phone is Tier 3, so you can use it as a lead but not as a published fact until you find or produce stronger evidence. | Tier | What you can do with it | Examples | | --- | --- | --- | | Tier 0, canonical primary | Cite it directly for facts and claims. | Official docs, repositories, model cards, papers, release posts, benchmark repos. | | Tier 1, primary adjacent | Use it for context. Verify exact claims elsewhere. | Maintainer talks, company blogs, official demos, standards docs. | | Tier 2, reproducible community evidence | Cite it when the method and environment are visible. | Independent benchmark logs, notebooks, issue threads with maintainer confirmation. | | Tier 3, discovery signal | Treat it as a lead. Do not publish it as fact. | X and Twitter posts, newsletters, podcasts, third party amplification. | | Tier 4, unsupported | Do not publish it as fact at all. | Search snippets, rumors, unattributed claims, private notes without approval. | ## Handle X and Twitter posts An X post can point you to a demo, a contributor question, or interest in the ecosystem. A post cannot carry a performance, license, device, or quality claim by itself. Trace the post back to a durable source before you publish anything from it. ```text X post -> source task -> durable source -> claim/source row -> public page ``` ## Promote a claim Before you move a claim onto a public page, answer these questions: - Does the claim have an official or reproducible source? - Is the source dated? - Does the claim name the model, runtime, device, and benchmark scope where relevant? - Does the public page preserve uncertainty, e.g., "we have not yet run this on hardware"? - Is the claim free of unapproved roadmap or product language? If the answer to any question is no, hold the claim back. ## Quarantine a claim Hold a claim out of public pages if any of these is true: - It comes only from social media. - It names performance without a benchmark setup. - It names a license without model card or repository license evidence. - It mentions a product roadmap or private notes. - It turns an open question into a fact. ## Next steps - Trace where published claims stand today in the [claim and source matrix](https://yourwildcard.ai/docs/prismml/claim-source-matrix.md). - Review incoming leads in the [X discovery notes](https://yourwildcard.ai/docs/sources/x-discovery-notes.md). - Learn how to submit a source in the [contributor guide](https://yourwildcard.ai/docs/contribute/contributor-guide.md). --- --- title: "X Discovery Notes: PrismML Signal Queue" status: published audience: Researcher owner: Docs librarian source_tier: discovery benchmark_status: not_applicable last_reviewed: 2026-07-07 canonical_url: https://yourwildcard.ai/docs/sources/x-discovery-notes --- # X discovery notes An X post about PrismML is a lead, not a fact. This page is the queue of every PrismML claim we found on X, each paired with the official source that could confirm or kill it, and here you can pick up the next claim to verify or add a new one in the same shape. None of the claims below has been checked against an official source yet. Until someone verifies a claim on the [source policy](https://yourwildcard.ai/docs/sources/source-policy.md) page, no other page on this wiki may repeat it as fact. The numbers on this page (1.21 GB, 0.93 GB, 1.15 GB, "4B", "8B", compression ratios, license terms) come from X posts. Do not cite any of them anywhere else until you have verified them against an official blog post, model card, or whitepaper. ## Verify a queued claim Each row below records one X post, the claim we saw in it, and the official source that can settle it. To verify a claim, open the sources in its "How to verify" cell, compare them against the observed claim, and record the result on the [source policy](https://yourwildcard.ai/docs/sources/source-policy.md) page. If the official source contradicts the post, mark the row here as "killed" with the contradicting URL and keep it as a record. For example, the first row queues a license claim. To close it, you would open the Bonsai Image 4B announcement and the Hugging Face model cards, confirm whether Apache 2.0 covers both the 1-bit and ternary weights, and record what you found. | X post | Claim as observed | How to verify | | --- | --- | --- | | [PrismML 2059339169806307422](https://x.com/PrismML/status/2059339169806307422) | 1-bit and ternary Bonsai Image 4B released under Apache 2.0, with blog, demo, model, and whitepaper links. | Check the license and release scope in the [Bonsai Image 4B announcement](https://prismml.com/news/bonsai-image-4b), the [demo repo](https://github.com/PrismML-Eng/Bonsai-image-demo), and the Hugging Face model cards. Confirm whether Apache 2.0 covers both weight variants or only one. | | [PrismML 2059339168250253731](https://x.com/PrismML/status/2059339168250253731) | Bonsai Studio iPhone app runs Bonsai Image 4B with on-device generation. | Find the official App Store listing and device requirements before anyone writes a build-and-run recipe for it. | | [PrismML 2059339166056632570](https://x.com/PrismML/status/2059339166056632570) | Compression framed as "retained usefulness," referencing composition, preference, aesthetics, and prompt-following quality. | Find a linked benchmark source for each quality claim. Without one, no wiki page may repeat the framing. | | [PrismML 2059339163061940524](https://x.com/PrismML/status/2059339163061940524) | Ternary Bonsai Image 4B described as a 1.21 GB ternary-weight model with a large footprint reduction versus full precision. | Check the exact file size and compression ratio against the blog, whitepaper, and model card, and note which full-precision baseline the reduction is computed against. | | [PrismML 2059339159899390326](https://x.com/PrismML/status/2059339159899390326) | 1-bit Bonsai Image 4B described as a 0.93 GB maximum-compression variant. | Check the exact size, supported runtimes, and license in the model cards. | | Vinod Khosla statuses (IDs not captured) | Compression presented as a way to cut energy and cost and widen access. Includes a claimed 1-bit Bonsai 8B at 1.15 GB. | Re-run the [broad search](https://x.com/search?q=%28PrismML%20OR%20%22Bonsai%208B%22%20OR%20%22Bonsai%20Image%22%29%20%28vkhosla%20OR%20Vinod%20OR%20Khosla%20OR%20Ion%20OR%20Stoica%20OR%20inference%20OR%20edge%29&src=typed_query&f=live), record the exact status URLs, then check the 1.15 GB figure against the [Bonsai 8B GGUF model card](https://huggingface.co/prism-ml/Bonsai-8B-gguf). | | [@eraznafre](https://x.com/eraznafre) EasyDeL preview (status ID not captured) | A development branch called vnext, a v0.4.0 release not yet on PyPI, new components named SpectraX, eRay, and ejKernel, pipeline parallelism with 9 schedulers, 9 new reinforcement learning trainers, and up to 32 percent faster than main on their benchmarks. | Wait for the v0.4.0 release and changelog on [EasyDeL on GitHub](https://github.com/erfanzar/EasyDeL) and [PyPI](https://pypi.org/project/easydel/). Check each named component against the released code and the docs at [easydel.readthedocs.io](https://easydel.readthedocs.io/). The stable v0.2.0.2 release and the eSurge engine are already visible in the repository and can be cited today. | | [@HessianFree](https://x.com/HessianFree) EasyDeL amplification (status ID not captured) | A person described as a PrismML co founder praises the EasyDeL vnext branch and says the team has used it for months. Implies EasyDeL and PrismML are the same team. | Find an official source that links EasyDeL to PrismML, e.g. a PrismML page, an EasyDeL README, or a shared release. Until then, do not state that EasyDeL and PrismML are one team. | We recorded the theme of the Khosla posts and the EasyDeL preview, but not their status IDs or the exact wording. The EasyDeL rows also mix a verifiable part, which is the stable release and the eSurge engine in the repository, with unverified parts, which are the vnext components and the 32 percent number. Cite only the repository part until someone archives the post URLs and a v0.4.0 release lands. ## Add a new discovery We collected these signals on 2026-07-06 with a read-only browser pass on X. We made no likes, reposts, follows, bookmarks, replies, or posts. To add new discoveries, repeat that pass with the two saved searches and keep it read-only: - [PrismML broad search](https://x.com/search?q=%28PrismML%20OR%20%22Bonsai%208B%22%20OR%20%22Bonsai%20Image%22%29%20%28vkhosla%20OR%20Vinod%20OR%20Khosla%20OR%20Ion%20OR%20Stoica%20OR%20inference%20OR%20edge%29&src=typed_query&f=live) - [Official PrismML search](https://x.com/search?q=from%3APrismML%20%28Bonsai%20OR%20%221-bit%22%20OR%20ternary%20OR%20local%20OR%20inference%20OR%20edge%20OR%20WebGPU%29&src=typed_query&f=live) Record each new signal as a table row with three things: 1. The exact X post URL, captured at discovery time. 2. The claim exactly as you observed it, without cleanup or interpretation. 3. The official source that can confirm or kill it. When you later cite a verified claim on another page, cite the official source, never the X post. Never turn third-party enthusiasm, such as amplification, benchmark screenshots, or size comparisons, directly into a project claim. ## Read the third-party themes Beyond the official PrismML posts, third-party posts fall into four themes. These themes tell you where community interest sits, and none of them is evidence for any claim. - **Local image generation.** AlphaSignalAI and Rahul (`@codecroc`) amplified Bonsai Image 4B as local image generation on phones, browsers, and WebGPU. - **Edge inference.** Third-party posts framed PrismML around edge computing and the difficulty of serving one request at a time efficiently. - **Architecture efficiency.** A Jon Durbin thread asked what architecture maximizes both training and inference efficiency and pointed at MoE-style tradeoffs. - **Cost and energy.** Khosla posts and PrismML launch materials present compression as a way to cut energy and cost and widen access. ## Next steps - [Source policy](https://yourwildcard.ai/docs/sources/source-policy.md). Read how a claim moves from a lead to a published fact, and record your verification results there. - [Contributor guide](https://yourwildcard.ai/docs/contribute/contributor-guide.md). Start here if this is your first contribution to the source pipeline. --- --- title: DevRel Operating Model status: published audience: DevRel owner: DevRel maintainer source_tier: internal_canonical benchmark_status: not_applicable last_reviewed: 2026-07-07 canonical_url: https://yourwildcard.ai/docs/governance/operating-model --- # Operating model The operating model is the set of roles, steps, and schedules that keep this wiki accurate. Use this page when you need to decide who owns a piece of incoming work, how to check it, and when it is ready to publish. Incoming work is a page, a recipe, a benchmark, a source update, or an ecosystem entry. Every kind of work follows the same path here: it gets an owner, a reviewer, a source check, and an acceptance test before it goes live. ## Know the roles Each role below owns one kind of decision. One person can hold more than one role, but every published page needs a named page owner and a named reviewer. For example, a new llama.cpp recipe has a recipe author who writes it, a runtime validator who repeats the setup, and a reviewer who signs off before the maintainer publishes it. | Role | Owns | Main outputs | | --- | --- | --- | | DevRel maintainer | Wiki roadmap, page set, publication decisions | Accepted backlog, published pages, contributor guide | | Docs librarian | Source quality, claim tracking, link hygiene | Claim/source matrix, source status, quarantined claims list | | Researcher | Discovery work and source collection | Research notes, official source fallbacks, open questions | | Recipe author | Local runtime and integration guides | Runnable recipes with acceptance tests | | Benchmark owner | Test schema, task suite, result quality | Benchmark template, recorded runs, regression notes | | Page owner | One page or one ecosystem entry | Drafts, refresh dates, issue triage for that page | | Layer owner | One supply chain layer | Routing decisions, entry consistency, open gaps | | Runtime validator | Runtime checks for local recipes | Repeated setup results, device notes, failed paths | | Reviewer | Practical check before publication | Source review, runnable check, clarity review | | Ecosystem liaison | Upstream wording and validation | PrismML or project confirmations, roadmap links, approved wording | A claim is quarantined when we remove it from reader pages because it has no primary source. It stays on the quarantined list until someone finds an official source for it or retires it. ## Take in new work Every new piece of work goes through the same eight steps before anyone starts writing: 1. Capture the request in the backlog or issue tracker. 2. Assign the work to one supply chain layer from the [ecosystem map](https://yourwildcard.ai/docs/ecosystem/supply-chain-layers.md). 3. Assign the work to one public wiki page. 4. Classify the source state as primary, discovery only, mixed, or unsourced. 5. Name the page owner, reviewer, and source owner. 6. Define the output as a doc, recipe, benchmark artifact, or source update. 7. Define the acceptance test before work starts. 8. Reject or quarantine any product claim that has no primary source. Record the answers on an intake card. A filled card for a recipe request looks like "Title: Bonsai with Ollama. Work type: recipe. Acceptance test: a reviewer can pull the model and get one completion." Copy this template: ```markdown ### Intake card - Title: - Work type: page | recipe | benchmark | source update | ecosystem entry - Target page: - Supply chain layer: - Contributor: - Page owner: - Reviewer: - Source owner: - Source state: primary | discovery only | mixed | unsourced - Expected output: - Acceptance test: - Open questions: - Publish blocker: ``` ## Route requests to the right place Match the request to its topic and send it to the page that owns that topic. Each route also names the proof the work needs before it can publish. | If the request is about | Route it to | Required proof | | --- | --- | --- | | PrismML model claims, file size, license, device support, or quality | The [PrismML claim/source matrix](https://yourwildcard.ai/docs/prismml/claim-source-matrix.md) | Official PrismML post, repo, model card, paper, or app page | | llama.cpp, MLX, Ollama, LM Studio, WebGPU, or iPhone paths | Runtime recipe backlog, e.g. [Bonsai with llama.cpp](https://yourwildcard.ai/docs/build-and-run/bonsai-llamacpp.md) | A command or documented manual path that a reviewer can repeat | | Quality, latency, memory, energy, or offline behavior | Benchmark harness pages | Benchmark schema and recorded run details | | Architecture, KV cache, kernels, or training methods | Explainers under [Technical Guides](https://yourwildcard.ai/docs/technical-guides/kv-cache.md) | Primary docs, papers, or official project docs | | X/Twitter signal or third-party amplification | Research notes first | Durable official source before publication. Claims from X stay labeled "discovery, needs primary source" | ## Move pages through the lifecycle A page moves through nine states from proposal to publication. Each state has one exit rule, and the person named in the rule decides when the page moves forward. For example, a draft recipe cannot reach Runnable checked until someone repeats its setup on a real machine. | State | Exit rule | | --- | --- | | Proposed | Intake card has owner, reviewer, source state, output, and acceptance test | | Accepted | Maintainer confirms the page belongs in the wiki architecture | | Draft | Page owner writes the first version and links all sources | | Source checked | Docs librarian verifies claims and marks gaps | | Runnable checked | Recipe author or reviewer repeats the setup, prompt, benchmark, or manual flow | | Reviewed | Reviewer checks clarity, audience fit, source status, and acceptance test | | Published | DevRel maintainer approves and assigns refresh date | | Refresh due | Page owner checks links, source status, and current runtime behavior | | Retired or quarantined | Maintainer removes stale or unsupported public claims | A recipe that no one has run stays labeled "not verified" and must ship with an acceptance test a reviewer can repeat, e.g. "A reviewer can start a local OpenAI-compatible server and run one prompt." It cannot pass Runnable checked on the author's word. ## Assign owners to pages Every page area has a default owner, a required reviewer, and an event that triggers a refresh. When you take on a page, check this table to see who reviews your work and what event means the page needs another look. | Page | Primary owner | Required reviewer | Refresh trigger | | --- | --- | --- | --- | | Orientation | DevRel maintainer | Docs librarian | Thesis or map changes | | Small-model state of the art | Researcher | Docs librarian | New model family, benchmark, or architecture source | | Runtime and KV cache | Runtime page owner | Recipe author or benchmark owner | Runtime release or recipe change | | Training and alignment | Training page owner | Benchmark owner | New fine-tuning or alignment path | | PrismML integration guide | PrismML liaison | Docs librarian | PrismML release, model card change, or app update | | Benchmark harness | Benchmark owner | DevRel maintainer | New metric, task suite, or published result | | Contribution guide | DevRel maintainer | External contributor reviewer | Contributor friction or workflow change | | Runtime recipes | Recipe author | Runtime validator | Runtime, model file, device, or command change | | Ecosystem entries | Page owner | Layer owner | Source change or owner change | Every maintained page records its ownership in the page frontmatter or the issue tracker. Record the page owner, reviewer, source owner, supply chain layer, last source check, last runnable check, next refresh date, and current status. Readers never see this record, so keep it out of the page body. ## Keep pages current Run these checks on a schedule so pages do not go stale between releases. | Cadence | Task | | --- | --- | | Weekly | Triage new intake cards and assign owners | | Weekly | Review source hygiene blockers and quarantined claims | | Every release or model update | Refresh PrismML pages and runtime recipes | | Monthly | Check priority backlog items and retire stale work | | Before publication | Complete the acceptance test and ownership record | ## Write and review to the standard Apply these rules before writing or reviewing any page: 1. Name the reader by role, not by vague interest. 2. State the reader's task, e.g. run, compare, or verify. 3. Classify the topic under exactly one supply chain layer in the frontmatter, and keep that classification out of the reader prose. 4. Convert social or search discoveries into source tasks before publishing claims. 5. Prefer runnable proof and benchmark logs over broad assertions. 6. State uncertainty in plain reader language, e.g. "These commands come from the upstream docs; we have not yet run them on hardware." 7. End every page with a next step the reader can click. ## Next steps - Check the sourcing rules your work must meet in the [source policy](https://yourwildcard.ai/docs/sources/source-policy.md). - Trace how product claims get verified in the [PrismML claim/source matrix](https://yourwildcard.ai/docs/prismml/claim-source-matrix.md). - Send new contributors to the [contributor guide](https://yourwildcard.ai/docs/contribute/contributor-guide.md). --- --- title: DevRel Playbooks status: published audience: DevRel owner: DevRel maintainer source_tier: mixed benchmark_status: not_applicable last_reviewed: 2026-07-07 canonical_url: https://yourwildcard.ai/docs/governance/devrel-playbooks --- # DevRel playbooks A DevRel playbook is a short, repeatable procedure for a recurring community task, such as announcing a release or preparing a live demo. This page gives you four playbooks for PrismML work. You can write a launch brief, run office hours, check a demo before you give it, and route community feedback to the right owner. Nobody has run these playbooks yet. The steps and success metrics below are our best proposal, not a record of what worked. If you run one, tell us what happened so we can correct it. ## Write a launch brief When PrismML ships a new model, updates a model card, or changes a runtime fork, write a brief that the community team can publish within 48 hours. The brief answers launch-day questions with sourced facts only. Before you publish, check every claim against the [claim and source matrix](https://yourwildcard.ai/docs/prismml/claim-source-matrix.md). The matrix is the table where we record each public claim about PrismML next to the official source that backs it. A claim with no matching official source stays out of the brief and goes through the [source policy](https://yourwildcard.ai/docs/sources/source-policy.md) instead. For example, if PrismML releases a new Bonsai build, your brief lists the release date, links the official repo and model card, describes what changed in one plain paragraph, and states which wiki pages and recipes the change touches. It also lists what we do not claim yet, so an advocate on stage knows where the line is. Use this template: ```markdown ### Launch brief - Release name and date: - Official sources (repo / model card / company post URLs): - What changed (one paragraph, no adjectives without sources): - Claims added to matrix (link rows): - Affected wiki pages and recipes: - Runtime/device support changes (source URL per claim): - Talking points a DA can say without overclaiming: - Things we do NOT claim yet (open questions): - Demo asset to use (see the demo checklist below): - Owner / reviewer / publish date: ``` The PrismML liaison owns the brief and the docs librarian reviews it. During launch week, turn every question the brief cannot answer into a backlog card so the next brief can answer it. The brief succeeds when every published claim has a verified or partially verified matrix row. ## Run office hours Hold a live session where contributors ask questions about recipes, benchmarks, and ecosystem entries. A Developer Advocate hosts, and the recipe author or benchmark owner joins when the agenda touches their work. We suggest a weekly session, plus an extra one after each launch brief, but the source docs do not fix a cadence, platform, or time zone yet. Give every question one of three exits: - Answer it with a link to a wiki page. - Convert it into a backlog card with a named owner. - Log it as an open question on the page it belongs to. The session succeeds when every question has a recorded exit. If the same question comes back a second time, fix the page instead of repeating the answer. Use these standing assets instead of improvising material: - [Inference The Hard Way](https://chaiwithjai.github.io/inference-the-hard-way/) walks through tokenization, KV cache, decode, and bandwidth. Use it when you explain how PrismML Bonsai inference works. We confirmed the link works on 2026-07-06. - [Kubernetes The Hard Way](https://bootstrap-your-own-k8s.netlify.app) covers serving and cluster questions. The source repo is [github.com/ChaiWithJai/kubernetes-the-hard-way](https://github.com/ChaiWithJai/kubernetes-the-hard-way). - The [Bonsai llama.cpp recipe](https://yourwildcard.ai/docs/build-and-run/bonsai-llamacpp.md) is the target for live runs and the device report walkthrough. ## Check a demo before you give it Do not ship a demo that a reviewer could not reproduce from the wiki alone. Before any talk, office-hours session, or partner meeting that includes a live run, work through this checklist. The recipe author prepares the demo and the runtime validator signs off. ```markdown ### Demo readiness checklist - [ ] Recipe page exists and its commands come from checked upstream sources - [ ] Runtime built from a named commit or release - [ ] Model file downloaded from an official model card URL - [ ] Environment record filled (device, OS, RAM/VRAM, runtime commit) - [ ] Dry run completed within 48h of the session; TTFT and tokens/sec noted - [ ] Offline mode tested if the demo claims local/offline operation - [ ] Fallback: recording or screenshots for the failure case - [ ] Every number said aloud has a claim-matrix row or is labeled "my machine, one run" ``` The environment record block lives in the [Bonsai llama.cpp recipe](https://yourwildcard.ai/docs/build-and-run/bonsai-llamacpp.md). If a checklist item fails, fix the recipe or add a troubleshooting entry before the next demo. The demo succeeds when it runs from documented commands at a named runtime commit, and when you have stated, not assumed, how it behaves offline. A demo is a benchmark claim delivered out loud. Latency, tokens per second, and memory figures you say in a session follow the same source rules as published pages. Each figure needs a claim-matrix row, or you say plainly that it comes from a single run on your machine. ## Route community feedback When a question, bug report, benchmark dispute, or social post comes in, send it to the owner who can act on it. The Community Manager routes each signal, and the weekly triage in the [operating model](https://yourwildcard.ai/docs/governance/operating-model.md) reviews what was routed. A signal that sits unrouted for more than a week goes to the DevRel maintainer. For example, if someone posts on X that Bonsai runs at 40 tokens per second on an iPhone, do not publish that number. Record it as a discovery note, then look for a run a reviewer can repeat. Nothing we publish rests only on an X or Twitter post. Route each signal by what it is about: | Signal is about | Route to | Required proof before publication | | --- | --- | --- | | PrismML model claims, file size, license, device support | PrismML pages and the claim/source matrix | Official PrismML post, repo, model card, paper, or app page | | llama.cpp, MLX, Ollama, LM Studio, WebGPU, iPhone paths | Runtime recipe backlog | A command a reviewer can repeat | | Quality, latency, memory, energy, offline behavior | Benchmark harness | Benchmark schema and recorded run details | | Architecture, KV cache, kernels, training methods | Concept explainers | Primary docs or papers | | X/Twitter signal or third-party amplification | Research notes first | A durable official source | Routing succeeds when every signal lands with an owner within one weekly triage cycle, and when no published claim rests only on a social post. ## Run one first If you want to put these playbooks to work, start with the demo checklist. Run it against the [Bonsai llama.cpp recipe](https://yourwildcard.ai/docs/build-and-run/bonsai-llamacpp.md) on one device and attach the filled environment record and checklist to an issue titled "First demo-readiness run: Bonsai llama.cpp". That single run also produces the first device report the recipe needs. ## Next steps - Read the [source policy](https://yourwildcard.ai/docs/sources/source-policy.md) to see the rules every brief and demo figure must pass. - Check the [claim and source matrix](https://yourwildcard.ai/docs/prismml/claim-source-matrix.md) before you publish any claim. - See the [contributor guide](https://yourwildcard.ai/docs/contribute/contributor-guide.md) for how to file cards and log a playbook run. --- --- title: Page Types, Front Matter, and Contribution Templates status: published audience: Page owner owner: DevRel maintainer source_tier: internal_note benchmark_status: not_applicable last_reviewed: 2026-07-07 canonical_url: https://yourwildcard.ai/docs/governance/page-types-and-templates --- # Page types and templates Every page in this wiki is one of seven page types, and each type comes with a fixed front matter block and a template. Use this page to pick the type for a page you are about to write, copy the matching front matter, and fill in the ownership fields before you publish. ## Pick a page type A page type says what job a page does for the reader. A page holds exactly one type. If your draft mixes two, e.g., a concept explanation that also walks through install commands, split it into two pages. For example, "Bonsai 8B with llama.cpp" is a runtime recipe. It exists so a reader can run the model, so it must list the device, the runtime, the install steps, the exact command, and the output the reader should expect. See [/docs/build-and-run/bonsai-llamacpp](https://yourwildcard.ai/docs/build-and-run/bonsai-llamacpp.md) for the reference shape. A recipe that no reviewer has run yet says so in plain words on the page, e.g., "These commands come from the upstream docs; we have not yet run them on hardware." | Page type | Use when | Required content | | --- | --- | --- | | Orientation | A reader needs context before choosing a task. | Audience, prerequisite knowledge, map location, next three links. | | Concept guide | A technical idea needs an explanation independent of one vendor. | Definition, why it matters, tradeoffs, diagrams, canonical sources, related recipes. | | Runtime recipe | A reader should be able to run something. | Device, OS, runtime, model, install steps, command, expected output, troubleshooting, benchmark notes. | | Ecosystem entry | A project needs a stable place in the map. | Official URL, layer, contribution, install status, benchmark status, source tier, owner. | | Benchmark report | A result should be comparable over time. | Model, runtime, device, prompt set, metrics, raw logs, date, reproduction notes, caveats. | | DevRel playbook | A team needs a repeatable community workflow. | Goal, audience, trigger, artifacts, owner, feedback loop, success metric. | | Source note | A social or news signal needs triage before it becomes canonical. | Discovery URL, source tier, claim, primary-source fallback, status, next action. | ## Copy the front matter Front matter is the YAML block at the top of each page file. Readers never see it. Maintainers and tooling use it to filter, review, and refresh pages by field. Start every new page from this example and change the values. ```yaml title: "Bonsai 8B with llama.cpp" page_type: "runtime_recipe" audience: ["devrel_engineer", "developer_advocate"] ecosystem_layer: ["runtime_serving", "compression_formats"] status: "draft" # seed | draft | verified | canonical | needs_review | retired owner: "unassigned" last_reviewed: "2026-07-06" source_tier: "primary_confirmed" benchmark_status: "not_run" supported_runtimes: ["llama.cpp"] supported_devices: ["apple_silicon", "linux_cuda"] claim_matrix: "./benchmarks/claim-source-matrix.md" related_pages: - "../ecosystem-map/prismml.md" - "../concepts/kv-cache.md" ``` Follow these field rules: - `ecosystem_layer` lists exactly one primary layer. Add secondary layers only when they help contributors route work. - `source_tier` uses the tiers defined in the [source policy](https://yourwildcard.ai/docs/sources/source-policy.md). The tiers are primary confirmed, primary social, secondary technical, secondary social, and internal note. - `benchmark_status` is one of `not_run`, `partial`, `reproducible`, `published`, or `not_applicable`. A result stays `partial` until another reviewer repeats it. - `owner` is a role from the operating model, e.g., Page owner or Benchmark owner. It is never "unassigned" at publication. ## Fill the owner block The owner block is a short checklist near the top or bottom of each maintained page. It records who maintains the page and where it sits in the maintenance workflow, which the front matter deliberately does not track. Copy it as is and fill in each line. ```markdown ## Page ownership - Page owner: - Reviewer: - Source owner: - Primary supply chain layer: - Benchmark owner, if any: - Last source check: - Last runnable check: - Next refresh: - Current status: draft | source checked | runnable checked | published | refresh due | quarantined ``` Every published page needs a named page owner and a named reviewer. One person can hold both roles on different pages, but not on the same page. ## File an intake card An intake card is a short form you file before starting any new page, recipe, benchmark, source update, or ecosystem entry. It forces two decisions up front. You define the acceptance test before work starts, and you name a primary source for every product claim. A claim with no primary source is rejected or quarantined at intake. Quarantined means the claim is held out of published pages until someone finds a primary source for it. ```markdown ### Intake card - Title: - Work type: page | recipe | benchmark | source update | ecosystem entry - Target page: - Supply chain layer: - Contributor: - Page owner: - Reviewer: - Source owner: - Source state: primary | discovery only | mixed | unsourced - Expected output: - Acceptance test: - Open questions: - Publish blocker: ``` ## Use the three status vocabularies correctly The wiki runs three separate status vocabularies on purpose. Each one answers a different question, and mixing them hides risk. For example, a page can be trustworthy as a whole (`verified` in front matter) while one new claim on it is still `discovery`. If both facts shared one label, you could not see the risky claim. | Vocabulary | Lives in | Question it answers | | --- | --- | --- | | Page lifecycle (`seed`, `draft`, `verified`, `canonical`, `needs_review`, `retired`) | Front matter `status` | Is this page trustworthy as a whole? | | Operational state (`draft`, `source checked`, `runnable checked`, `published`, `refresh due`, `quarantined`) | Owner block `Current status` | Where is this page in the maintenance workflow? | | Uncertainty labels (`discovery`, `partial`, `not verified`, `stale`, `quarantined`) | Inline on claims and recipes | Can this specific claim or command be trusted? | The three vocabularies connect through gates: - **Claims gate the owner block.** A page cannot reach `source checked` while any claim on it is `discovery` or `stale`. It cannot reach `runnable checked` while a recipe on it is `not verified`. If any claim is `quarantined`, the whole page stays `quarantined` until you remove the claim. - **The owner block gates the front matter.** `verified` in front matter requires `source checked`, and for recipes it also requires `runnable checked`. `canonical` requires `published` plus maintainer review. `refresh due` in the owner block maps to `needs_review` in front matter. - **Promotion is never automatic.** Fixing one stale claim does not promote a page on its own. Promotion always goes through the reviewer named in the owner block. Tools and reviewers filter on these exact strings. A page marked `checked` or `done` is invisible to the review queue. Copy values verbatim from the tables above. ## Next steps - Read the [source policy](https://yourwildcard.ai/docs/sources/source-policy.md) to learn how source tiers are assigned before you fill in `source_tier`. - Check the [claim and source matrix](https://yourwildcard.ai/docs/prismml/claim-source-matrix.md) to see how claims are traced to primary sources. - Return to the [contributor guide](https://yourwildcard.ai/docs/contribute/contributor-guide.md) for the overall contribution workflow. --- --- title: Wiki Evaluation Rubric status: published audience: Wiki reviewer owner: Docs librarian source_tier: internal_canonical benchmark_status: not_applicable last_reviewed: 2026-07-07 canonical_url: https://yourwildcard.ai/docs/governance/evaluation-rubric --- # Evaluation rubric The evaluation rubric is the scoring standard that decides whether a page in this wiki is ready to publish. Use it to score a submitted page, check the hard gates, and assign one review outcome. ## Score the six dimensions Score each dimension from 0 to 3. The maximum total is 18, and a publishable page normally scores at least 15 with no hard gate failures. For example, a page that names its reader, prerequisites, and desired outcome scores 3 on stated audience. If the same page lists commands the author never ran without saying so, it scores 1 on runnable paths and also fails a hard gate. | Dimension | 0 | 1 | 2 | 3 | | --- | --- | --- | --- | --- | | Stated audience | No reader or job named | Generic audience only | Reader and use case named | Reader, prerequisites, desired outcome, and non-goals are clear | | Canonical source hygiene | Unsourced or social-only claims | Some sources, weak provenance | Primary sources used for major claims | Claim, source, date, and status are traceable, and X is discovery only | | Runnable paths | No run path or verification state | High-level instructions only | Commands or workflow included, with gaps marked | Minimal path, expected result, environment, and failure notes are reproducible | | Visual comprehension | No map or table where needed | Visual exists but does not clarify | Table or diagram clarifies the layer or flow | Visual makes the system easier to review and labels its assumptions | | Contribution usefulness | No next action | Vague "contribute" language | Specific tasks or gaps listed | Issue-ready work with output, owner type, acceptance test, or template fields | | Benchmark and eval clarity | Quality claims are untested | Metrics named without protocol | Benchmark status and metrics are explicit | Task, dataset or prompt set, device and runtime, quality, latency, memory, energy, and regression checks are clear | ### Read the total as a band | Total | Band | Action | | --- | --- | --- | | 16 to 18 | Canonical | Publish or promote as a reference page. | | 13 to 15 | Useful draft | Publish only if hard gates pass and TODOs are explicit. | | 9 to 12 | Contributor note | Keep in the backlog or draft area. | | 0 to 8 | Not ready | Rework around sources, audience, and verification. | ## Check the hard gates first A hard gate is a rule that fails the page no matter how high it scores. Check the gates before you score, because scoring a page that fails a gate wastes your time and risks publishing unverified public claims. Fail the page until fixed if any item is true: 1. A product, benchmark, compression, licensing, device, or performance claim lacks a primary source and is not marked unverified. 2. X posts, search snippets, or third-party enthusiasm are treated as canonical evidence. 3. A recipe implies it was run but lacks device, runtime, version, or output details. 4. Benchmark language mixes marketing claims with measured results. 5. Open questions are presented as facts. 6. Unapproved or rumored claims, such as unreleased product names, appear in public-facing copy. If any gate fails, assign the quarantine label or return the page to the author without scoring it. ## Work through the review checklist Score the dimensions in this order. **Audience and scope.** The first section says who the page is for and what task it helps with, such as learn, install, compare, benchmark, verify, or contribute. **Sources.** Every important claim has a source URL, a date, and a verification status. Official docs, repos, model cards, papers, and company posts count more than social posts. Social links stay as discovery context and are paired with durable official sources. Source gaps are listed separately from confirmed facts. **Runnable or inspectable path.** Commands, setup steps, model and runtime versions, hardware assumptions, and expected output are present when the page has a run path. If the author did not run the path, the page says so plainly, e.g., "These commands come from the upstream docs; we have not yet run them on hardware", and explains what evidence is missing. See [the Bonsai llama.cpp recipe](https://yourwildcard.ai/docs/build-and-run/bonsai-llamacpp.md) for a page that does this well. **Visual and structural clarity.** Tables summarize ecosystem entries, runtime compatibility, or device results. Diagrams explain the supply chain flow or system boundaries. Each visual should remove ambiguity, not decorate the page. **Contribution value.** The page gives the reader a concrete next step. A new contributor can submit a source update, recipe, benchmark run, or correction without asking where to start. **Benchmark and eval.** Benchmark status is one of `not_run`, `partial`, `reproducible`, or `published`. `not_run` means no one has run the benchmark yet. Metrics match the deployment target, e.g., latency and memory for an on-device page. Evaluation claims name the task, prompt set or dataset, runtime, hardware, model variant, quantization, date, and regression expectation. ## Rate sources by tier When you audit the source hygiene dimension, place each source in a tier. A lower tier number means stronger evidence. | Tier | Use | Examples | | --- | --- | --- | | Tier 0: Canonical primary | Strong basis for facts and claims | Official docs, repositories, model cards, papers, release posts | | Tier 1: Primary-adjacent | Useful context. Verify exact claims elsewhere. | Maintainer talks, company blogs, official demos | | Tier 2: Reproducible community evidence | Useful if the method and environment are visible | Independent benchmark logs, runnable notebooks, issue threads with maintainer confirmation | | Tier 3: Discovery signal | Good lead, not canonical | X posts, newsletters, podcasts | | Tier 4: Unsupported | Do not publish as fact | Search snippets, rumors, unattributed claims | ## Confirm the community value checks Before you publish, confirm each statement is true: 1. A new contributor can identify the intended reader and a useful next action in under five minutes. 2. A maintainer can trace every important claim to a durable source. 3. A builder can run, reproduce, or knowingly skip the documented path. 4. A reviewer can tell what is measured, what is assumed, and what remains unverified. 5. The page improves the shared ecosystem map instead of duplicating generic ML background. 6. The page produces a durable artifact, such as a doc, a runnable recipe, a benchmark result, or a contribution template. ## Assign one outcome label End every review with exactly one label. | Label | Meaning | | --- | --- | | `canonical` | Passes the hard gates and scores 16 to 18. | | `publish-with-todos` | Passes the hard gates, scores 13 to 15, and lists its remaining gaps. | | `draft` | Useful but incomplete. Visible to contributors only, with gaps made explicit. | | `quarantine` | Contains unverified public claims, source-risk issues, or misleading benchmark language. Fix before any further review. | ## Next steps - Apply the tiers in detail with the [source policy](https://yourwildcard.ai/docs/sources/source-policy.md). - See how a scored review fits the full workflow in the [operating model](https://yourwildcard.ai/docs/governance/operating-model.md). - Submitting a page yourself? Start with the [contributor guide](https://yourwildcard.ai/docs/contribute/contributor-guide.md). --- --- title: Blog canonical_url: https://yourwildcard.ai/blog --- # All posts Ternary Bonsai 27B is the main story here. Twenty-six focused posts connect the public checkpoint and whitepaper to local M4 Pro traces, runtime choices, tool calling, reasoning budgets, fine tuning, and the launch conversation. The broader inference archive follows below. ## Ternary Bonsai 27B Start with the [field guide](https://yourwildcard.ai/docs/prismml/bonsai-27b.md) or the [ecosystem roundup centered on The Information](/blog/bonsai-27b-ecosystem-reactions). - [What is Ternary Bonsai 27B?](/blog/what-is-ternary-bonsai-27b) - [What Mac do you need for Ternary Bonsai 27B?](/blog/ternary-bonsai-27b-mac-requirements) - [How to run Ternary Bonsai 27B with MLX](/blog/run-ternary-bonsai-27b-mlx) - [How to serve it through an OpenAI-compatible API](/blog/ternary-bonsai-27b-openai-api) - [How many bits per weight does it really use?](/blog/ternary-bonsai-27b-bits-per-weight) - [Binary versus ternary weights](/blog/binary-vs-ternary-bonsai-27b) - [Why the ideal and packaged model sizes differ](/blog/ternary-bonsai-27b-model-size) - [How the KV cache changes memory use](/blog/ternary-bonsai-27b-kv-cache) - [Measured M4 Pro performance](/blog/ternary-bonsai-27b-performance-m4-pro) - [What the quality benchmarks show](/blog/ternary-bonsai-27b-quality-benchmarks) - [Qwen3.6 27B versus Ternary Bonsai 27B](/blog/qwen36-27b-vs-ternary-bonsai-27b) - [Using Bonsai 27B with Hermes Agent](/blog/ternary-bonsai-27b-hermes-agent) - [Using Bonsai 27B in LM Studio](/blog/ternary-bonsai-27b-lm-studio) - [Whether ODS is a tenable runtime path](/blog/ternary-bonsai-27b-ods) - [How to approach fine tuning](/blog/fine-tune-ternary-bonsai-27b) - [Known limits](/blog/ternary-bonsai-27b-limitations) - [The benchmark method](/blog/ternary-bonsai-27b-benchmark-method) - [Why reasoning-token budgets matter](/blog/ternary-bonsai-27b-reasoning-token-budget) - [How prompt caching changes the result](/blog/ternary-bonsai-27b-prompt-cache) - [How to resume an interrupted download](/blog/ternary-bonsai-27b-download-resume) - [What we learned on a 24 GB Mac](/blog/ternary-bonsai-27b-24gb-memory) - [The structured tool-call test](/blog/ternary-bonsai-27b-tool-calling-test) - [Why speed changes between runs](/blog/ternary-bonsai-27b-runtime-variance) - [How to compare vendor and local benchmarks](/blog/ternary-bonsai-27b-vendor-vs-local-benchmarks) - [How to verify the exact checkpoint](/blog/ternary-bonsai-27b-model-provenance) - [What the ecosystem is saying about Bonsai 27B](/blog/bonsai-27b-ecosystem-reactions) ## Foundational concepts The ideas under every inference decision, from the KV cache to quantization. - [Can 1-bit models do tool calling?](/blog/can-1-bit-models-do-tool-calling) - [Why does batch size trade latency for throughput in LLM serving?](/blog/batch-size-latency-throughput-tradeoff) - [How much VRAM do I need to run an LLM?](/blog/how-much-vram-to-run-an-llm) - [How does speculative decoding generate more than one token per forward pass?](/blog/how-speculative-decoding-works) - [How does the KV cache make attention linear instead of quadratic?](/blog/kv-cache-linear-attention) - [Why is model selection the biggest inference optimization?](/blog/model-selection-biggest-inference-optimization) - [Why is LLM prefill compute-bound but decode memory-bound?](/blog/prefill-compute-bound-decode-memory-bound) - [Should I use a shared LLM API or a dedicated deployment?](/blog/shared-llm-api-vs-dedicated-deployment) - [What are the three layers of an inference stack?](/blog/three-layers-of-an-inference-stack) - [How do I verify a quantized model hasn't lost quality?](/blog/verify-quantized-model-quality) - [What are prefill and decode in LLM inference?](/blog/what-are-prefill-and-decode) - [What do TTFT and TPS actually measure for LLMs?](/blog/what-do-ttft-and-tps-measure) - [What does 'intelligence density' actually measure — and why isn't it just tokens per second?](/blog/what-is-intelligence-density) - [Why do API providers charge less for cached input tokens?](/blog/why-cached-input-tokens-cost-less) - [Why are GPUs faster than CPUs for AI inference?](/blog/why-gpus-beat-cpus-for-inference) - [Why does quantization make LLM inference faster in both prefill and decode?](/blog/why-quantization-speeds-up-inference) - [Why should I report P99 latency instead of average?](/blog/why-report-p99-latency-not-average) - [How should product teams think about accuracy vs. speed trade-offs in AI?](/blog/accuracy-vs-speed-tradeoffs-ai-products) - [What's the difference between fine-tuning and distillation?](/blog/fine-tuning-vs-distillation) - [How many GPUs do I need to serve a large LLM?](/blog/how-many-gpus-to-serve-an-llm) - [Why do images blow up my LLM's context window?](/blog/images-blow-up-context-window) - [Inference is fast but my app feels slow — what do I check?](/blog/inference-fast-but-app-feels-slow) - [Is generative AI the same thing as AI?](/blog/is-generative-ai-the-same-as-ai) - [How do I make model cold starts faster?](/blog/make-model-cold-starts-faster) - [What metrics should I monitor for an LLM inference service?](/blog/metrics-to-monitor-llm-inference) - [What's the difference between online and offline inference?](/blog/online-vs-offline-inference) - [How should I order my prompt to maximize prefix cache hits?](/blog/prompt-ordering-for-prefix-cache-hits) - [Which parts of an LLM are safest to quantize: weights, activations, KV cache, or attention?](/blog/safest-parts-of-llm-to-quantize) - [Tensor vs pipeline vs expert parallelism: which should I use for LLM inference?](/blog/tensor-vs-pipeline-vs-expert-parallelism) - [Do inference optimizations like quantization and speculation stack together?](/blog/do-inference-optimizations-stack) - [Why are MoE models fast locally but not on servers?](/blog/moe-fast-locally-slow-on-servers) - [What determines whether speculative decoding actually speeds things up?](/blog/what-determines-speculative-decoding-speedup) - [What is disaggregated serving (prefill/decode separation) and when is it worth it?](/blog/what-is-disaggregated-serving) - [What is an ops:byte ratio and why does it matter for GPUs?](/blog/what-is-ops-byte-ratio) - [What makes long context expensive, and what are chunked prefill and paged KV for?](/blog/what-makes-long-context-expensive) - [Why are floating-point formats better than integers for LLM quantization?](/blog/why-floating-point-beats-integer-quantization) - [Why is image generation compute-bound when the models are small?](/blog/why-image-generation-is-compute-bound) ## Ecosystem players The runtimes, engines, and platforms that serve small models, compared honestly. - [Does the 1-bit Bonsai whitepaper prove the efficiency thesis this wiki is built on?](/blog/bonsai-whitepaper-and-the-efficiency-thesis) - [What do FlashAttention and PagedAttention actually optimize?](/blog/flashattention-vs-pagedattention) - [How do I benchmark an LLM server properly?](/blog/how-to-benchmark-an-llm-server) - [Ollama vs llama.cpp — which should I use for local inference?](/blog/ollama-vs-llama-cpp) - [vLLM vs SGLang vs TensorRT-LLM — which inference engine should I pick?](/blog/vllm-vs-sglang-vs-tensorrt-llm) - [Why is Apple Silicon good for running LLMs locally?](/blog/why-apple-silicon-runs-llms-well) - [What's the fastest way to run Whisper locally — streaming or batch?](/blog/fastest-way-to-run-whisper-locally) - [Should I use safetensors or ONNX for my model?](/blog/safetensors-vs-onnx) - [What are DeepSeek-R1 distilled models and should I run one locally?](/blog/what-are-deepseek-r1-distilled-models) - [What is a MIG (multi-instance GPU) and when should I use one?](/blog/what-is-a-mig-fractional-gpu) - [What is GGUF block quantization and how do Q4_K models actually work?](/blog/what-is-gguf-block-quantization) - [When is NVIDIA Dynamo worth the complexity?](/blog/when-is-nvidia-dynamo-worth-it) - [Who is your developer ecosystem actually for? End users vs ecosystem members](/blog/who-is-your-developer-ecosystem-for) - [What's the difference between benchmarking and profiling inference?](/blog/benchmarking-vs-profiling-inference) - [What content do runtimes, MSPs, and inference studios need that end users never read?](/blog/content-for-runtimes-msps-inference-studios) - [When should I use a draft model vs EAGLE vs n-gram speculation?](/blog/draft-model-vs-eagle-vs-ngram-speculation) - [Why should I pin exact dependency versions in my inference container?](/blog/pin-dependency-versions-inference-containers) - [How do I serve embeddings for both bulk indexing and live search?](/blog/serving-embeddings-bulk-vs-live) - [What does 'sparse FLOPS' mean on a GPU spec sheet, and should I trust it?](/blog/should-i-trust-sparse-flops) - [What is a CUDA kernel in simple terms?](/blog/what-is-a-cuda-kernel) - [What is cache-aware routing and why does round-robin load balancing hurt LLM latency?](/blog/what-is-cache-aware-routing) - [Why do GPU kernels break when you upgrade GPUs?](/blog/why-gpu-kernels-break-on-upgrades) - [Why don't prefix caching and tensor parallelism help embedding models?](/blog/why-llm-optimizations-dont-help-embeddings) - [Why do multi-GPU LLM deployments need NVLink or InfiniBand?](/blog/why-multi-gpu-needs-nvlink-or-infiniband) ## End-user case studies How real teams in regulated and cost-sensitive settings adopt local AI. - [What does a hospital CTO actually ask before running a model on-prem?](/blog/hospital-cto-on-prem-ai-questions) - [Should I run AI on-device or in the cloud? The eight-factor tradeoff](/blog/on-device-vs-cloud-ai-inference) - [At what token volume does running my own GPU beat a per-token API?](/blog/token-volume-where-own-gpu-beats-api) - [What does an AI product manager actually do day to day?](/blog/what-does-an-ai-product-manager-do) - [How many clicks to first inference? Auditing your docs like a funnel](/blog/clicks-to-first-inference-docs-funnel) - [When is GenAI the wrong tool for personalization?](/blog/genai-wrong-tool-for-personalization) - [What belongs in a local-AI runbook for a compliance-bound team?](/blog/local-ai-runbook-for-compliance-teams) - [Should I use a speech-to-speech model or an ASR-LLM-TTS pipeline?](/blog/speech-to-speech-vs-asr-llm-tts-pipeline) - [How can I transcribe an hour of audio in seconds?](/blog/transcribe-an-hour-of-audio-in-seconds) - [When is scale-to-zero a bad idea for AI apps?](/blog/when-scale-to-zero-is-a-bad-idea) - [Why do worked examples beat case studies in regulated industries?](/blog/worked-examples-beat-case-studies-regulated) - [What's the difference between AGI hype and today's AI capabilities?](/blog/agi-hype-vs-shipping-reality) - [Are AI agents just chatbots with tools?](/blog/are-ai-agents-just-chatbots-with-tools) - [What is speech recognition vs. text-to-speech, in plain terms?](/blog/asr-tts-plain-terms-offline-voice-assistant) - [How do you benchmark a local model when your budget is a power outlet, not a GPU cluster?](/blog/benchmarking-on-a-power-outlet-budget) - [Do I need an LLM for ticket routing, or is a classifier enough?](/blog/classifier-or-llm-for-ticket-routing) - [How do I compare per-token API costs to renting my own GPUs?](/blog/compare-api-costs-to-renting-gpus) - [Why do data-rich companies have an AI advantage?](/blog/data-advantage-without-an-llm) - [When should I use a few-step image model?](/blog/few-step-image-models-for-realtime-features) - [Is a bigger model ever cheaper? Rethinking cost per unit of capability](/blog/is-a-bigger-model-ever-cheaper) - [Should I use one omni-modal model or a pipeline of small specialists?](/blog/ocr-model-vs-vlm-for-receipts) - [Should you run the demo before you finish the paper?](/blog/run-the-demo-before-finishing-the-paper) - [How do safety frameworks translate into an on-prem AI deployment checklist?](/blog/safety-framework-to-onprem-checklist) - [Why should every demo be extendable, not just runnable?](/blog/should-every-demo-be-extendable) - [What if your UI taught users how inference works?](/blog/ui-that-teaches-how-inference-works) - [What legitimate uses do deepfakes have, and how should teams manage the risk?](/blog/voice-cloning-consent-and-watermarking) - [Why do voice assistants struggle with accents, and whose job is it to fix?](/blog/why-voice-assistants-struggle-with-accents) - [Will every product manager become an AI product manager?](/blog/will-every-pm-become-an-ai-pm) --- --- title: Contributor Guide status: published audience: External contributor owner: DevRel maintainer source_tier: internal_operating_model benchmark_status: not_applicable last_reviewed: 2026-07-07 canonical_url: https://yourwildcard.ai/docs/contribute/contributor-guide --- # Contributor guide A contribution to this wiki is a documented fact about small local models or the tools that run them, backed by a source a reviewer can check. This page gives you the checklist a reviewer will apply to your change, so you can pass review on the first attempt. You can contribute four kinds of work: - an ecosystem entry, which describes one project such as a runtime or a training tool - a runtime recipe, which shows how to run a model on a specific device - a benchmark result, which records what you measured on that device - a PrismML claim, which is one sourced fact about the PrismML model family If you do not have a change in mind yet, pick a task from [the backlog](https://yourwildcard.ai/docs/contribute/backlog.md) and come back here before you open a pull request. ## Meet the definition of done A reviewer marks a page as done when all nine items are true: 1. It has an official URL, and a repository, model card, paper, or docs URL when one exists. 2. It is assigned to exactly one primary supply chain layer, which tells reviewers who owns it. 3. It lists secondary layers only when they help route work. 4. Every public claim has a source and a date checked. 5. Install or run status is marked as verified, partial, not verified, or not applicable. 6. Benchmark status is marked as not run, partial, reproducible, or published. 7. Open questions are listed separately from facts. 8. A page owner and a reviewer are named. 9. The next refresh date is listed. A reviewer rejects or quarantines any product claim that has no primary source. A quarantined claim is held out of public docs until someone finds a source for it. An X/Twitter link counts as discovery only. It can point us at a claim, but it cannot support one. A published claim needs an official doc, repo, paper, model card, company post, or benchmark artifact behind it. ## Submit an ecosystem entry - [ ] Add the official URL. - [ ] Add the repository, model card, paper, or docs URL when one exists. - [ ] Pick one primary supply chain layer. - [ ] Say what the project contributes in one or two sentences. - [ ] Mark install or run status. - [ ] Mark benchmark status. - [ ] List open questions separately from facts. - [ ] Name the owner. ## Submit a runtime recipe - [ ] List the model file, runtime, device, operating system, and source URL. - [ ] Include exact setup steps. - [ ] Include one command or manual flow that a reviewer can repeat. - [ ] Record latency, tokens per second, memory, file size, and offline behavior when available. - [ ] Mark unsupported devices and untested paths. - [ ] Include an acceptance test, e.g. "A reviewer can start a local OpenAI-compatible server and run one prompt." If you have not personally run the recipe, say so in the recipe. The reviewer labels it not verified until someone records a run on hardware. See [Bonsai with llama.cpp](https://yourwildcard.ai/docs/build-and-run/bonsai-llamacpp.md) for the reference shape, including the environment record block. ## Submit a benchmark result - [ ] State the task, prompt set, model, runtime, quantization, device, and date. - [ ] Record time to first token, tokens per second, memory, and power when available. - [ ] Keep quality notes separate from speed and memory results. - [ ] Link the benchmark script, schema, or manual protocol. - [ ] Mark the result as partial unless another reviewer can repeat it. ## Submit a PrismML claim Every public statement about PrismML lives as one row in the [claim/source matrix](https://yourwildcard.ai/docs/prismml/claim-source-matrix.md). For example, the claim "Bonsai 8B ships under an Apache 2.0 license" would appear as a row that names the field checked (license), links the official model card, and records the date you checked it. - [ ] Add the claim to the claim/source matrix before it appears in public docs. - [ ] Link the official PrismML source or model card. - [ ] Record the exact field checked, such as license, file size, supported runtime, or device support. - [ ] Mark X/Twitter-only claims as discovery. - [ ] Remove or quarantine the claim when no primary source exists. Each row in the matrix carries one of five statuses. The [claim/source matrix](https://yourwildcard.ai/docs/prismml/claim-source-matrix.md) is the source of truth for these definitions: - verified. An official source states the fact directly and a reviewer confirmed it on the source, or we have run and benchmark evidence. - partial. A source exists, but it does not cover the full claim, so quote it with the caveat in the notes. - source-linked. A durable source exists, but a reviewer still needs the exact quote or version. - discovery. The signal comes only from social posts or search, so do not treat it as fact. - quarantined. Do not publish this as fact. Only verified claims and partial claims with a clear scope appear in public docs. ## Pick a first task Pick from [the backlog](https://yourwildcard.ai/docs/contribute/backlog.md). Higher priority does not mean harder work, and the P0 items are often the fastest first contributions. Every backlog item ends in one of four outputs: a durable doc, a runnable recipe, a benchmark artifact, or a source update. If your idea does not map to one of those, refine it with a maintainer first. ## Know who reviews your work Each contribution type has one reviewer role, and the table below shows what that reviewer checks. Full role definitions live in the [operating model](https://yourwildcard.ai/docs/governance/operating-model.md). | Your contribution | Reviewed by | What they check | | --- | --- | --- | | Ecosystem entry | Layer owner | Layer assignment, entry consistency, open gaps. | | Runtime recipe | Runtime validator | Repeated setup, device notes, failed paths. | | Benchmark result | Benchmark owner | Schema fit, recorded run details, repeatability. | | PrismML claim | Docs librarian | Claim/source matrix row, source tier, date checked. | | Any page before publication | Reviewer, then DevRel maintainer | Clarity, audience fit, source status, acceptance test; final publish and refresh date. | One person can hold more than one role, but every published page needs a named page owner and a named reviewer. Complete the definition of done and add the owner block, which names the page owner, the reviewer, the source owner, the primary layer, the last source check, the last runnable check, the next refresh date, and the current status. Maintainers triage new submissions weekly, so name your owner and reviewer when you submit rather than waiting for review to assign them. ## Next steps - [Pick a task from the backlog](https://yourwildcard.ai/docs/contribute/backlog.md) - [Read the source policy](https://yourwildcard.ai/docs/sources/source-policy.md) to see how we tier sources before you cite one. - [Read the operating model](https://yourwildcard.ai/docs/governance/operating-model.md) if you want to take on a reviewer role. --- --- title: Contribution Backlog status: published audience: Community maintainer owner: DevRel maintainer source_tier: internal_standard benchmark_status: not_applicable last_reviewed: 2026-07-07 canonical_url: https://yourwildcard.ai/docs/contribute/backlog --- # Contribution backlog The contribution backlog is the ordered list of work this wiki needs next, from hardware verification to new runtime recipes. Pick a task below, file it as an issue with the template on this page, and you have a scoped piece of work you can finish on your own. ## Pick a task Each task names what you produce and how we accept it. The acceptance test is the check a reviewer runs before the work merges, so read it before you start. For example, the first P0 task asks you to run llama.cpp on an Apple Silicon machine, record a device report, and update the recipe. It is accepted when another reviewer can reproduce one prompt from your notes. P0 tasks block other work, so take one of those first if you can. | Priority | Task | You produce | Accepted when | | --- | --- | --- | --- | | P0 | Verify llama.cpp on Apple Silicon. | Device report and recipe update. | Another reviewer can reproduce one prompt. | | P0 | Complete the PrismML claim/source matrix. | Dated source rows. | Every numeric claim has a durable source or is marked unverified. | | P1 | Add an MLX Bonsai recipe. | New recipe page. | Install, model, prompt, output, and failure notes are recorded. | | P1 | Add a runtime compatibility matrix. | Table covering llama.cpp, MLX, Ollama, LM Studio, WebGPU, and iOS. | Each row has a status and an evidence requirement. | | P1 | Add a KV cache visual. | Diagram or table showing cache growth. | A reader can calculate cache memory from architecture fields. | | P2 | Build the Inference Inspector spec. | Capstone spec page. | Phase bar, KV gauge, bandwidth ledger, and token stream are defined. | | P2 | Add a source queue workflow. | Intake template. | A contributor can submit a source task without a maintainer meeting. | Pick the task that matches your skills. The P0 verification task needs the hardware and a terminal. The claim/source matrix needs someone comfortable tracing citations. The recipe and compatibility tasks need a contributor who runs models locally, and the visual and spec tasks fit writers and demo builders. ## File the issue Copy this template into a new issue and fill in every heading. A blank heading usually means the task is not scoped yet, so ask in the issue rather than guessing. ```markdown ## Task ## Page route ## Audience job ## Source inputs ## Output artifact ## Acceptance test ## Reviewer ## Known gaps ``` ## Check your work before you ask for review Your task is done when all of these are true: - The page route exists. - The page frontmatter is updated. - Every claim on the page names its source, or says plainly that it is unverified. - The page says honestly whether anyone has run its commands on hardware. - Anything you could not resolve is written on the page, not left out. ## Next steps - Read the [contributor guide](https://yourwildcard.ai/docs/contribute/contributor-guide.md) for how review works end to end. - Check the [source policy](https://yourwildcard.ai/docs/sources/source-policy.md) before you add or cite a source. - See the [claim/source matrix](https://yourwildcard.ai/docs/prismml/claim-source-matrix.md) if you take the P0 sourcing task. --- --- title: Blog canonical_url: https://yourwildcard.ai/blog --- # Blog index Ternary Bonsai 27B is the main story here. Twenty-six focused posts connect the public checkpoint and whitepaper to local M4 Pro traces, runtime choices, tool calling, reasoning budgets, fine tuning, and the launch conversation. The broader inference archive follows below. ## Ternary Bonsai 27B Start with the [field guide](https://yourwildcard.ai/docs/prismml/bonsai-27b.md) or the [ecosystem roundup centered on The Information](/blog/bonsai-27b-ecosystem-reactions). - [What is Ternary Bonsai 27B?](/blog/what-is-ternary-bonsai-27b) - [What Mac do you need for Ternary Bonsai 27B?](/blog/ternary-bonsai-27b-mac-requirements) - [How to run Ternary Bonsai 27B with MLX](/blog/run-ternary-bonsai-27b-mlx) - [How to serve it through an OpenAI-compatible API](/blog/ternary-bonsai-27b-openai-api) - [How many bits per weight does it really use?](/blog/ternary-bonsai-27b-bits-per-weight) - [Binary versus ternary weights](/blog/binary-vs-ternary-bonsai-27b) - [Why the ideal and packaged model sizes differ](/blog/ternary-bonsai-27b-model-size) - [How the KV cache changes memory use](/blog/ternary-bonsai-27b-kv-cache) - [Measured M4 Pro performance](/blog/ternary-bonsai-27b-performance-m4-pro) - [What the quality benchmarks show](/blog/ternary-bonsai-27b-quality-benchmarks) - [Qwen3.6 27B versus Ternary Bonsai 27B](/blog/qwen36-27b-vs-ternary-bonsai-27b) - [Using Bonsai 27B with Hermes Agent](/blog/ternary-bonsai-27b-hermes-agent) - [Using Bonsai 27B in LM Studio](/blog/ternary-bonsai-27b-lm-studio) - [Whether ODS is a tenable runtime path](/blog/ternary-bonsai-27b-ods) - [How to approach fine tuning](/blog/fine-tune-ternary-bonsai-27b) - [Known limits](/blog/ternary-bonsai-27b-limitations) - [The benchmark method](/blog/ternary-bonsai-27b-benchmark-method) - [Why reasoning-token budgets matter](/blog/ternary-bonsai-27b-reasoning-token-budget) - [How prompt caching changes the result](/blog/ternary-bonsai-27b-prompt-cache) - [How to resume an interrupted download](/blog/ternary-bonsai-27b-download-resume) - [What we learned on a 24 GB Mac](/blog/ternary-bonsai-27b-24gb-memory) - [The structured tool-call test](/blog/ternary-bonsai-27b-tool-calling-test) - [Why speed changes between runs](/blog/ternary-bonsai-27b-runtime-variance) - [How to compare vendor and local benchmarks](/blog/ternary-bonsai-27b-vendor-vs-local-benchmarks) - [How to verify the exact checkpoint](/blog/ternary-bonsai-27b-model-provenance) - [What the ecosystem is saying about Bonsai 27B](/blog/bonsai-27b-ecosystem-reactions) ## Foundational concepts The ideas under every inference decision, from the KV cache to quantization. - [Can 1-bit models do tool calling?](/blog/can-1-bit-models-do-tool-calling) - [Why does batch size trade latency for throughput in LLM serving?](/blog/batch-size-latency-throughput-tradeoff) - [How much VRAM do I need to run an LLM?](/blog/how-much-vram-to-run-an-llm) - [How does speculative decoding generate more than one token per forward pass?](/blog/how-speculative-decoding-works) - [How does the KV cache make attention linear instead of quadratic?](/blog/kv-cache-linear-attention) - [Why is model selection the biggest inference optimization?](/blog/model-selection-biggest-inference-optimization) - [Why is LLM prefill compute-bound but decode memory-bound?](/blog/prefill-compute-bound-decode-memory-bound) - [Should I use a shared LLM API or a dedicated deployment?](/blog/shared-llm-api-vs-dedicated-deployment) - [What are the three layers of an inference stack?](/blog/three-layers-of-an-inference-stack) - [How do I verify a quantized model hasn't lost quality?](/blog/verify-quantized-model-quality) - [What are prefill and decode in LLM inference?](/blog/what-are-prefill-and-decode) - [What do TTFT and TPS actually measure for LLMs?](/blog/what-do-ttft-and-tps-measure) - [What does 'intelligence density' actually measure — and why isn't it just tokens per second?](/blog/what-is-intelligence-density) - [Why do API providers charge less for cached input tokens?](/blog/why-cached-input-tokens-cost-less) - [Why are GPUs faster than CPUs for AI inference?](/blog/why-gpus-beat-cpus-for-inference) - [Why does quantization make LLM inference faster in both prefill and decode?](/blog/why-quantization-speeds-up-inference) - [Why should I report P99 latency instead of average?](/blog/why-report-p99-latency-not-average) - [How should product teams think about accuracy vs. speed trade-offs in AI?](/blog/accuracy-vs-speed-tradeoffs-ai-products) - [What's the difference between fine-tuning and distillation?](/blog/fine-tuning-vs-distillation) - [How many GPUs do I need to serve a large LLM?](/blog/how-many-gpus-to-serve-an-llm) - [Why do images blow up my LLM's context window?](/blog/images-blow-up-context-window) - [Inference is fast but my app feels slow — what do I check?](/blog/inference-fast-but-app-feels-slow) - [Is generative AI the same thing as AI?](/blog/is-generative-ai-the-same-as-ai) - [How do I make model cold starts faster?](/blog/make-model-cold-starts-faster) - [What metrics should I monitor for an LLM inference service?](/blog/metrics-to-monitor-llm-inference) - [What's the difference between online and offline inference?](/blog/online-vs-offline-inference) - [How should I order my prompt to maximize prefix cache hits?](/blog/prompt-ordering-for-prefix-cache-hits) - [Which parts of an LLM are safest to quantize: weights, activations, KV cache, or attention?](/blog/safest-parts-of-llm-to-quantize) - [Tensor vs pipeline vs expert parallelism: which should I use for LLM inference?](/blog/tensor-vs-pipeline-vs-expert-parallelism) - [Do inference optimizations like quantization and speculation stack together?](/blog/do-inference-optimizations-stack) - [Why are MoE models fast locally but not on servers?](/blog/moe-fast-locally-slow-on-servers) - [What determines whether speculative decoding actually speeds things up?](/blog/what-determines-speculative-decoding-speedup) - [What is disaggregated serving (prefill/decode separation) and when is it worth it?](/blog/what-is-disaggregated-serving) - [What is an ops:byte ratio and why does it matter for GPUs?](/blog/what-is-ops-byte-ratio) - [What makes long context expensive, and what are chunked prefill and paged KV for?](/blog/what-makes-long-context-expensive) - [Why are floating-point formats better than integers for LLM quantization?](/blog/why-floating-point-beats-integer-quantization) - [Why is image generation compute-bound when the models are small?](/blog/why-image-generation-is-compute-bound) ## Ecosystem players The runtimes, engines, and platforms that serve small models, compared honestly. - [Does the 1-bit Bonsai whitepaper prove the efficiency thesis this wiki is built on?](/blog/bonsai-whitepaper-and-the-efficiency-thesis) - [What do FlashAttention and PagedAttention actually optimize?](/blog/flashattention-vs-pagedattention) - [How do I benchmark an LLM server properly?](/blog/how-to-benchmark-an-llm-server) - [Ollama vs llama.cpp — which should I use for local inference?](/blog/ollama-vs-llama-cpp) - [vLLM vs SGLang vs TensorRT-LLM — which inference engine should I pick?](/blog/vllm-vs-sglang-vs-tensorrt-llm) - [Why is Apple Silicon good for running LLMs locally?](/blog/why-apple-silicon-runs-llms-well) - [What's the fastest way to run Whisper locally — streaming or batch?](/blog/fastest-way-to-run-whisper-locally) - [Should I use safetensors or ONNX for my model?](/blog/safetensors-vs-onnx) - [What are DeepSeek-R1 distilled models and should I run one locally?](/blog/what-are-deepseek-r1-distilled-models) - [What is a MIG (multi-instance GPU) and when should I use one?](/blog/what-is-a-mig-fractional-gpu) - [What is GGUF block quantization and how do Q4_K models actually work?](/blog/what-is-gguf-block-quantization) - [When is NVIDIA Dynamo worth the complexity?](/blog/when-is-nvidia-dynamo-worth-it) - [Who is your developer ecosystem actually for? End users vs ecosystem members](/blog/who-is-your-developer-ecosystem-for) - [What's the difference between benchmarking and profiling inference?](/blog/benchmarking-vs-profiling-inference) - [What content do runtimes, MSPs, and inference studios need that end users never read?](/blog/content-for-runtimes-msps-inference-studios) - [When should I use a draft model vs EAGLE vs n-gram speculation?](/blog/draft-model-vs-eagle-vs-ngram-speculation) - [Why should I pin exact dependency versions in my inference container?](/blog/pin-dependency-versions-inference-containers) - [How do I serve embeddings for both bulk indexing and live search?](/blog/serving-embeddings-bulk-vs-live) - [What does 'sparse FLOPS' mean on a GPU spec sheet, and should I trust it?](/blog/should-i-trust-sparse-flops) - [What is a CUDA kernel in simple terms?](/blog/what-is-a-cuda-kernel) - [What is cache-aware routing and why does round-robin load balancing hurt LLM latency?](/blog/what-is-cache-aware-routing) - [Why do GPU kernels break when you upgrade GPUs?](/blog/why-gpu-kernels-break-on-upgrades) - [Why don't prefix caching and tensor parallelism help embedding models?](/blog/why-llm-optimizations-dont-help-embeddings) - [Why do multi-GPU LLM deployments need NVLink or InfiniBand?](/blog/why-multi-gpu-needs-nvlink-or-infiniband) ## End-user case studies How real teams in regulated and cost-sensitive settings adopt local AI. - [What does a hospital CTO actually ask before running a model on-prem?](/blog/hospital-cto-on-prem-ai-questions) - [Should I run AI on-device or in the cloud? The eight-factor tradeoff](/blog/on-device-vs-cloud-ai-inference) - [At what token volume does running my own GPU beat a per-token API?](/blog/token-volume-where-own-gpu-beats-api) - [What does an AI product manager actually do day to day?](/blog/what-does-an-ai-product-manager-do) - [How many clicks to first inference? Auditing your docs like a funnel](/blog/clicks-to-first-inference-docs-funnel) - [When is GenAI the wrong tool for personalization?](/blog/genai-wrong-tool-for-personalization) - [What belongs in a local-AI runbook for a compliance-bound team?](/blog/local-ai-runbook-for-compliance-teams) - [Should I use a speech-to-speech model or an ASR-LLM-TTS pipeline?](/blog/speech-to-speech-vs-asr-llm-tts-pipeline) - [How can I transcribe an hour of audio in seconds?](/blog/transcribe-an-hour-of-audio-in-seconds) - [When is scale-to-zero a bad idea for AI apps?](/blog/when-scale-to-zero-is-a-bad-idea) - [Why do worked examples beat case studies in regulated industries?](/blog/worked-examples-beat-case-studies-regulated) - [What's the difference between AGI hype and today's AI capabilities?](/blog/agi-hype-vs-shipping-reality) - [Are AI agents just chatbots with tools?](/blog/are-ai-agents-just-chatbots-with-tools) - [What is speech recognition vs. text-to-speech, in plain terms?](/blog/asr-tts-plain-terms-offline-voice-assistant) - [How do you benchmark a local model when your budget is a power outlet, not a GPU cluster?](/blog/benchmarking-on-a-power-outlet-budget) - [Do I need an LLM for ticket routing, or is a classifier enough?](/blog/classifier-or-llm-for-ticket-routing) - [How do I compare per-token API costs to renting my own GPUs?](/blog/compare-api-costs-to-renting-gpus) - [Why do data-rich companies have an AI advantage?](/blog/data-advantage-without-an-llm) - [When should I use a few-step image model?](/blog/few-step-image-models-for-realtime-features) - [Is a bigger model ever cheaper? Rethinking cost per unit of capability](/blog/is-a-bigger-model-ever-cheaper) - [Should I use one omni-modal model or a pipeline of small specialists?](/blog/ocr-model-vs-vlm-for-receipts) - [Should you run the demo before you finish the paper?](/blog/run-the-demo-before-finishing-the-paper) - [How do safety frameworks translate into an on-prem AI deployment checklist?](/blog/safety-framework-to-onprem-checklist) - [Why should every demo be extendable, not just runnable?](/blog/should-every-demo-be-extendable) - [What if your UI taught users how inference works?](/blog/ui-that-teaches-how-inference-works) - [What legitimate uses do deepfakes have, and how should teams manage the risk?](/blog/voice-cloning-consent-and-watermarking) - [Why do voice assistants struggle with accents, and whose job is it to fix?](/blog/why-voice-assistants-struggle-with-accents) - [Will every product manager become an AI product manager?](/blog/will-every-pm-become-an-ai-pm) --- --- title: "How should product teams think about accuracy vs. speed trade-offs in AI?" description: "Accuracy versus speed is a product decision. The PM names the quality bar per feature, and the engineer finds the fastest configuration that clears it." audience: ml-product-team pillar: foundational-concept book: both chapter_ref: "Nika, Ch. 6 preview (chapter map); Kiely, Ch. 5, sec. 5.1" status: published last_reviewed: 2026-07-07 canonical_url: https://yourwildcard.ai/blog/accuracy-vs-speed-tradeoffs-ai-products --- # How should product teams think about accuracy vs. speed trade-offs in AI? Accuracy versus speed is a product decision expressed through technical dials, chiefly model size and quantization level. The PM names the quality bar for each feature, because a chat reply tolerates more error than a legal summary. The engineer then finds the fastest configuration that clears that bar. Neither role can set the operating point alone. ## Name the quality bar per feature The mistake teams make is treating accuracy and speed as one global setting for the whole product. They are per-feature settings. A casual chat feature can absorb an occasional clumsy sentence, because the user reads the reply, shrugs, and rephrases. A legal summary cannot absorb a wrong clause, because the user acts on it without checking the source. Same model, same company, two different quality bars. Naming the bar is product work, not engineering work. Marily Nika's Building AI-Powered Products treats model development (Ch. 6 in her chapter map) as a phase the PM stays inside, not one the PM hands off. The PM's part of the trade-off is a sentence like "for this feature, an error costs the user X, so the model must be right at least Y percent of the time on our eval set." Without that sentence, the engineer has no target and will optimize for whatever benchmark is handy. ## Know the dials the engineer turns Once the bar is named, the engineer has two main dials: - **Model size.** A smaller model reads fewer weights per token, so it produces tokens faster and costs less to serve. It is also, on average, less capable. - **Quantization level.** Storing the weights in fewer bits makes the same model faster. Kiely's Inference Engineering (Ch. 5, sec. 5.1) reports that dropping a single level of precision generally offers "30 to 50 percent better performance" for LLMs, and that the risk is that quantization "can materially reduce the model's output quality." Kiely's standard for production quantization is "zero perceptible quality loss" (sec. 5.1.3), verified by comparing the quantized model against the original on perplexity, a public benchmark, and your own evals. Note what that standard requires. Someone has to define what "perceptible" means for each feature, and that someone is the PM. The engineer measures the loss. The PM says whether the user would notice or care. Both dials also come in smaller steps than on or off. Kiely points out that quantization is a spectrum, e.g., you can quantize only the weights and leave the attention layers at full precision. So the frontier of available operating points is dense, and the question is never "fast or accurate." It is "which point on the frontier does this feature need." ## Work through the arithmetic Here is why the speed side of the dial is so large, using derived arithmetic rather than a benchmark. Token generation is limited by memory bandwidth, because the hardware reads the model weights once per token. An 8B model at 16-bit precision stores about 16 GB of weights. The same model at 4-bit stores roughly 4.5 GB once you include the format's overhead. That is about 3.5 times less data to read per token, so as a rough estimate the 4-bit version can generate tokens around 3 times faster on the same hardware. The exact number depends on the machine, so treat 3x as an estimate to verify, not a spec. Now translate that into user consequences. Suppose the 16-bit model generates 12 tokens per second on your hardware and a typical reply is 150 tokens. The reply takes 150 divided by 12, which is about 12.5 seconds. At 3 times the speed, the 4-bit version takes about 4 seconds. For a chat feature, 12.5 seconds is an abandoned conversation and 4 seconds is usable, so if the 4-bit model's errors stay within the chat bar, you ship 4-bit. For a legal summary that a lawyer waits 30 seconds for anyway, the 8 saved seconds buy the user almost nothing, so you spend the speed budget on accuracy and serve the higher precision. The public [Bonsai 8B model card](https://huggingface.co/prism-ml/Bonsai-8B-gguf) is a concrete example of the small end of this frontier. It exists because some features clear their quality bar with a small model, and for those features the extra speed and lower cost are free wins. ## Pick the operating point together The handoff between the two roles looks like this. The PM writes the quality bar per feature and names the user consequence behind it. The engineer builds the frontier, meaning a short table of configurations with measured accuracy and measured tokens per second for the task. Then both look at the table and pick a row per feature. The PM alone would pick a row without knowing what it costs to serve. The engineer alone would pick a row without knowing what an error does to the user. The row is a joint decision, and it should be written down with one line of rationale so the next person who touches the config knows why it is set where it is. Do not let the rationale be a benchmark number. "Q4 because MMLU only dropped 0.4" explains nothing to the next PM. "Q4 because chat users abandon after 5 seconds and our eval showed no visible reply quality change" survives a re-read a year later. ## Try it This takes under 30 minutes with llama.cpp. No GPU is required, because llama.cpp runs on a CPU or on Apple Silicon. Absolute speeds will be lower than on a GPU server, but the shape of the trade-off is the same. Download one small model at three quantization levels: ```bash brew install llama.cpp curl -LO https://huggingface.co/Qwen/Qwen2.5-0.5B-Instruct-GGUF/resolve/main/qwen2.5-0.5b-instruct-q2_k.gguf curl -LO https://huggingface.co/Qwen/Qwen2.5-0.5B-Instruct-GGUF/resolve/main/qwen2.5-0.5b-instruct-q4_k_m.gguf curl -LO https://huggingface.co/Qwen/Qwen2.5-0.5B-Instruct-GGUF/resolve/main/qwen2.5-0.5b-instruct-q8_0.gguf ``` Pick one task, e.g., summarize the same three paragraphs of a contract. Run it at each level and note the tokens per second that llama.cpp prints: ```bash llama-cli -m qwen2.5-0.5b-instruct-q2_k.gguf -f your-prompt.txt llama-cli -m qwen2.5-0.5b-instruct-q4_k_m.gguf -f your-prompt.txt llama-cli -m qwen2.5-0.5b-instruct-q8_0.gguf -f your-prompt.txt ``` Score each output yourself against a simple rubric, e.g., how many of the five key facts survived. Plot accuracy against tokens per second, which gives you a three-point frontier. Mark the point you would ship for a chat feature and the point you would ship for a legal summary, with one line of rationale each. The rationale must name a user consequence, not a score. ## Check yourself - Who names the quality bar, and who finds the configuration? The PM names the bar per feature, because it depends on what an error costs the user. The engineer finds the fastest configuration that clears it. - Your two chosen operating points came out identical. What went wrong? Either the two features truly have the same error cost, which is rare, or you scored against a generic rubric instead of each feature's user consequence. Recheck the rationale lines. - A teammate says "we should just always run Q8 to be safe." What is the counterargument? Safety is per feature. For the chat feature, Q8's extra latency is a cost users feel on every message, and it buys quality below the bar the feature needs. - Does each of your two rationale lines name a user consequence rather than a benchmark number? If one cites only a score, rewrite it. ## Next steps - Read the [quantization guide](https://yourwildcard.ai/docs/technical-guides/quantization.md) for what each level actually changes in the model. - Read the [intelligence density concept page](https://yourwildcard.ai/docs/concepts/intelligence-density.md) for how model size relates to capability per byte. - Before you ship a lower precision, run the acceptance test in [How do I verify a quantized model hasn't lost quality?](/blog/verify-quantized-model-quality). When you can do this, you can construct an accuracy and speed frontier for one task and select per-feature operating points with documented rationale. --- --- title: "What's the difference between AGI hype and today's AI capabilities?" description: "AGI is a frontier Nika dates to the 2030s with a question mark, not a shipped capability. Honest roadmaps separate what ships today from speculative ability." audience: ml-product-team pillar: end-user-case-study book: building-ai-products chapter_ref: "Nika, Ch. 1, 'Artificial General Intelligence (2030s?)'" status: published last_reviewed: 2026-07-07 canonical_url: https://yourwildcard.ai/blog/agi-hype-vs-shipping-reality --- # What's the difference between AGI hype and today's AI capabilities? The difference is shipped narrow capability against speculative general capability. Marily Nika's Building AI-Powered Products (Ch. 1) dates artificial general intelligence to the 2030s with a question mark, and she describes multidomain problem solving and faster R&D as its promise, not as anything a current product does. Copy that implies AGI sets expectations your product will visibly miss, and regulated buyers read overpromise as a risk signal. ## Read the question mark in the chapter heading Nika's section heading is "Artificial General Intelligence (2030s?)". The question mark is part of the claim. She calls AGI "the next frontier in AI research" and writes plainly that "we are not there yet" (Building AI-Powered Products, Ch. 1). Everything she lists under AGI is written in the future tense. AGI "will be able to tackle complex, multidomain problems", could speed up scientific discovery by generating hypotheses and running experiments, and could turn today's assistants into systems that manage large parts of daily life. Read that list next to what any current product does. A current product does one job, or a small set of jobs, inside a domain someone chose and tested. That is true of a fraud model and it is true of an LLM assistant. The assistant covers more surface than the fraud model, but it is still a specific system with tested strengths and known failure modes, not a general one. The chapter's future tense is the honest tense for AGI, and copy in the present tense should describe only what shipped. ## Sort marketing claims into three tiers A useful audit assigns every claim on a product page to one of three tiers. - Traditional. The claim describes one specific task done with rules or learned patterns, e.g., "flags duplicate invoices" or "transcribes calls". - GenAI. The claim describes content created from a prompt that a person then uses or reviews, e.g., "drafts a reply you approve". - Implies AGI. The claim describes ability across open-ended domains with no stated limits, e.g., "understands your business", "thinks like your best analyst", or "handles anything your customers ask". The first two tiers describe things a team can ship and demo today. The third tier borrows from Nika's 2030s section. The tell is a missing boundary. A shipped feature has a scope you can state, and copy in the third tier refuses to state one. The [intelligence density](https://yourwildcard.ai/docs/concepts/intelligence-density.md) page makes the same point from the engineering side, because a model has a measurable amount of capability per unit of compute, and copy should not promise more than the model holds. ## Work the arithmetic on a broken promise Overpromising is not only a tone problem. You can count what it costs. Take a support assistant trained on six intents, e.g., billing, refunds, and password resets. Suppose it handles 1,000 tickets a month. The landing page says it "handles anything your customers ask", which is a third tier claim, so users send it everything. Assume 20 percent of tickets fall outside the six trained intents. That share is an estimate, not a measurement, and your own logs will give you the true number. The arithmetic that follows is derived from that assumption. - Out-of-scope tickets per month: 1,000 times 0.20 is 200. - Out-of-scope tickets per year: 200 times 12 is 2,400. Each of those 2,400 tickets is a moment where the copy promised an answer and the product visibly failed to give one. Now rewrite the claim to match the product. "Resolves your six most common ticket types and hands the rest to your team" describes the same system. The 2,400 events still happen, but each one is now a handoff the copy predicted instead of a failure the copy denied. The product did not change. The promise did, and the promise is what the user grades. Regulated buyers grade the promise hardest. A hospital or bank buyer reads a third tier claim as evidence that the vendor either does not know its system's limits or will not say them, and both readings kill the deal. The [regulated deployment case study](https://yourwildcard.ai/docs/case-studies/regulated-deployment.md) shows the flip side, where a vendor won by stating scope and failure handling up front. ## Rewrite the claim without losing the sale Honest copy still has to sell. The rewrite above works because it kept the strongest true fact, which is that the assistant resolves the most common tickets on its own. If your honest rewrite reads as boring, the problem is usually that you grounded the adjectives and kept a weak claim, instead of finding the strongest claim the product can keep. "Drafts a compliant denial letter in 40 seconds, and a reviewer signs every one" is a first or second tier claim, it names a boundary, and it is more persuasive than "understands insurance" because a buyer can test it. ## Try it This takes about 25 minutes with a browser and a text editor. No GPU and no model download are needed. 1. Pick one product landing page, yours or a competitor's, and copy every capability claim into a list. Ten minutes. 2. Tag each claim traditional, GenAI, or implies AGI, using the definitions above. The test for the third tag is whether the claim states a boundary. Five minutes. 3. Take one claim tagged as implying AGI and rewrite it to describe the shipped capability, including what happens outside its scope. Five minutes. 4. Check the rewrite against the self test. It should still be compelling. If honesty made the sentence boring, you kept a weak claim and grounded its adjectives, so go find the strongest claim the product can keep and write that instead. Five minutes. If you want a grounded claim to practice on, the [Bonsai 8B model card](https://huggingface.co/prism-ml/Bonsai-8B-gguf) is a public example of first and second tier writing, because it states what the model is and what hardware runs it rather than what it understands. ## Check yourself - **A page says the assistant "learns your entire business in a day". Which tier?** Expected answer: it implies AGI. It claims open-ended ability across domains with no stated boundary, which Nika's chapter places in the 2030s with a question mark. - **A page says the tool "drafts release notes from merged pull requests". Which tier?** Expected answer: GenAI. It creates content from defined inputs, and the scope is stated. - **Why do regulated buyers punish claims that imply AGI harder than consumer buyers do?** Expected answer: because they must document system limits for their own regulators, so a vendor that overstates capability either does not know its limits or hides them, and both are disqualifying. - **Your honest rewrite is dull. What went wrong?** Expected answer: you grounded a weak claim instead of replacing it. The fix is to find the strongest claim the product can keep, with its boundary stated, and lead with that. ## Next steps - Read the [regulated deployment case study](https://yourwildcard.ai/docs/case-studies/regulated-deployment.md) to see scoped claims winning a sale under heavy compliance review. - Read the [intelligence density](https://yourwildcard.ai/docs/concepts/intelligence-density.md) page for how to reason about how much capability a model of a given size can hold. - For the four groups of AI that this three tier audit builds on, read [is generative AI the same thing as AI](/blog/is-generative-ai-the-same-as-ai). When you can do this, you can classify marketing claims by capability tier and rewrite overreach without losing persuasive force. --- --- title: "Are AI agents just chatbots with tools?" description: "No. Agentic behavior is a systems property built from a loop, tools, and memory, and a small local model in a ReAct loop shows exactly where it breaks." audience: researcher pillar: end-user-case-study book: building-ai-products chapter_ref: "Nika, front matter chapter map, Ch. 8 summary" status: published last_reviewed: 2026-07-07 canonical_url: https://yourwildcard.ai/blog/are-ai-agents-just-chatbots-with-tools --- # Are AI agents just chatbots with tools? No. Agentic behavior is a systems property built from a loop, tools, and memory, and it is separate from raw model scale. That makes it testable on small local models. A 1 to 2 GB model in a ReAct loop with one tool shows you exactly where the loop breaks, and that is more instructive than watching a frontier model succeed. ## Separate the loop from the model Marily Nika's Building AI-Powered Products gives Chapter 8 to agents, and the front matter chapter map says the chapter covers autonomous agents, multi-agent systems, and their use in real products. The chapter has a section titled "Not Just Glorified Chatbots", and its argument is a decomposition, not a size claim. Nika writes that ChatGPT on its own is not an agent because it "needs explicit input, lacks a goal-driven framework, and doesn't act within an environment" (Ch. 8). She lists the parts that make a system an agent: abilities, goals, prior knowledge, stimuli, and past experiences. For a researcher, the useful reading of that list is that every part except one lives outside the model weights. - The loop is code you write. It calls the model, parses a tool call out of the reply, runs the tool, and appends the result to the transcript. - The tool is code you write, plus the format the model must emit to call it. - The memory is the transcript itself, which grows every turn. - The model is the only part that changes when you change scale. So agentic behavior is a property of the whole system, and you can hold the loop, the tool, and the memory fixed while you swap the model. That turns "is this an agent problem or a model problem" from an opinion into an ablation. A ReAct loop is the simplest version of this system. The model writes a short reasoning step, then either a tool call or a final answer, the loop runs the tool and appends the result, and the cycle repeats. ## Work one task through the loop Here is the worked example. Give the system one tool, a calculator, and this task: "What is 12.5 percent of the sum of 384 and 416?" The correct trace has two tool calls. First the model calls the calculator with 384 + 416 and reads back 800. Then it calls the calculator with 800 * 0.125 and reads back 100. Then it answers 100. You can check the arithmetic by hand, since 384 + 416 = 800 and one eighth of 800 is 100. Now count what the model must carry. As a rough estimate, a system prompt that explains the tool format takes about 300 tokens, and each turn adds about 120 tokens of reasoning, tool call, and tool result. By the final answer the model is reading about 300 + 3 * 120 = 660 tokens of history, and it must keep three facts straight inside that history: the original question, the first result (800), and which step it is on. When I run this task with a 1.5B parameter model, the failures cluster at turn two. The model reuses 800 as the final answer, or it emits a tool call with a missing quote so the parser rejects it, or it recomputes 384 + 416 again because it lost track of which step it was on. A frontier model on the same loop almost always produces the clean two-call trace. That contrast is the data. The small model did not fail randomly. It failed at state tracking, at tool call formatting, and at recovery after a rejected call, and each of those points at a different part of the system. ## Sort each failure into one of three causes Each failure in your log belongs to one of three causes, and each cause has a different fix. - Loop design. The loop fed the wrong thing back, let the model answer before the tool returned, or had no retry after a bad parse. The fix is loop code. Adding one retry with the parser error shown to the model often recovers the formatting failures on its own. - Tool interface. The format you demanded is hard for the model to emit, e.g., strict nested JSON. The fix is a simpler format, such as a single line like `CALC: 384 + 416`. - Model capability. The loop is correct and the format is simple, and the model still loses the thread. The fix is a bigger or better model, and nothing you do to the loop will patch it. A frontier model hides the first two causes because it recovers from awkward formats and sloppy loops. The small model surfaces them, which is why the small model is the better instrument for studying the loop itself. ## Try it This takes about 25 minutes. No GPU is needed. A 1.5B model quantized to 4 bits is about 1 GB and runs on a laptop CPU. 1. Install llama.cpp and start a local server with a small model: ```bash brew install llama.cpp llama-server -hf Qwen/Qwen2.5-1.5B-Instruct-GGUF:Q4_K_M --port 8080 ``` 2. Hand-roll the loop. Do not use an agent framework, because the point is to own every line. The loop is about 30 lines of Python: post the transcript to `http://localhost:8080/v1/chat/completions`, look for a line starting with `CALC:` in the reply, evaluate the expression, append the result as a new message, and repeat up to 5 turns. Your system prompt defines the `CALC:` format and tells the model to write `ANSWER:` when done. 3. Run the percentage task from the worked example 10 times and log every transcript. 4. Swap in a larger model on the same loop and rerun. [Bonsai 8B](https://huggingface.co/prism-ml/Bonsai-8B-gguf) works as the larger local model, and as a rough estimate it needs about 6 GB of memory on CPU. A hosted frontier model through an API also works if you have a key. 5. Label every failed transcript with one cause: loop design, tool interface, or model capability. The test for each label is the fix. If a loop change would have saved the run, it is loop design. If a simpler format would have, it is tool interface. Otherwise it is model capability. ## Check yourself - **Where did the small model fail most often?** Expected answer: at turn two, on state tracking or on tool call formatting, not on the arithmetic itself. The calculator does the arithmetic. - **Which failures disappeared when you added a retry that shows the parser error?** Expected answer: most of the formatting failures. Those were loop design failures, not model failures, even though they looked like model failures at first. - **Did the larger model succeed because it is more agentic?** Expected answer: no. It ran the same loop with the same tool and the same memory. It succeeded because the model part of the system is stronger, which is exactly the separation the exercise demonstrates. - **What would you change first to help the small model, and why?** Expected answer: the tool interface, because a simpler format removes a whole failure class for free, and model capability is the most expensive part to change. ## Next steps - Read the [llama.cpp ecosystem page](https://yourwildcard.ai/docs/ecosystem/llama-cpp.md) for what the runtime does and how the server flags work. - Read the [inference stack overview](https://yourwildcard.ai/docs/concepts/inference-stack.md) to see where the loop you wrote sits relative to the runtime and the model. - For choosing between the runtimes you just used, read [Ollama vs llama.cpp](/blog/ollama-vs-llama-cpp). When you can do this, you can decompose agent failures into systems causes and model causes using a small model testbed. --- --- title: "What is speech recognition vs. text-to-speech, in plain terms?" description: "Speech recognition turns speech into text and TTS is the inverse. Both run in 1 to 2 GB locally, so a full offline voice assistant fits on a laptop." audience: inference-engineer pillar: end-user-case-study book: both chapter_ref: "Nika, Ch. 1, 'Speech' subsection; Kiely, Ch. 6, sec. 6.4" status: published last_reviewed: 2026-07-07 canonical_url: https://yourwildcard.ai/blog/asr-tts-plain-terms-offline-voice-assistant --- # What is speech recognition vs. text-to-speech, in plain terms? Speech recognition turns human speech into text. Text-to-speech, or TTS, is the inverse and gives the computer a voice. Both are decades old and both now run in about 1 to 2 GB of memory on a laptop. Modern TTS is literally an LLM with audio tokens added to its vocabulary, so a whole offline voice assistant fits on your machine. ## Separate the two directions of the same loop Speech recognition, also called ASR for automatic speech recognition, takes audio in and produces text out. Nika's Building AI-Powered Products (Ch. 1) introduces it in the Traditional AI section, because voice assistants like Siri and Alexa have used it for years and the research goes back decades. Her definition of the inverse is just as short. TTS systems "give computers the ability to speak in a humanlike manner" (Nika, Ch. 1). Nika opened her career on a speech team, and her point is that the hard part was never the definition. The hard part was accents, speaker identity, and fitting the technology into a product people would use. Neither direction needs a giant model. Whisper is the standard open ASR model, and Kiely's Inference Engineering (sec. 6.3) notes that the largest Whisper is 1.55B parameters, which is small next to any chat LLM. TTS models are small too. Kiely (sec. 6.4) gives Orpheus TTS at three billion parameters and calls that the larger end for the category. ## See why modern TTS is an LLM Kiely (sec. 6.4) explains what changed in TTS. Modern open TTS models are fine-tuned LLMs. Orpheus TTS is derived from Llama 3.2 3B. The recipe is to expand the LLM's vocabulary with tens of thousands of encoded audio tokens, then train the model "on pairs of text inputs with tokenized audio outputs" (Kiely, sec. 6.4). The model then generates audio tokens the same way a chat model generates words, and a separate small audio decoder converts those tokens into a waveform you can play. This is why the whole field converged. ASR, the LLM, and TTS now share most of their architecture on the decoder side, so the same runtimes and the same optimizations serve all three. It also changes the metrics. Kiely measures TTS with time to first byte, time to first sentence, and tokens per second, and he notes that a TTS model only needs on the order of 80 to 100 tokens per second to produce audio in real time. Faster decoding than that does not make the voice better. It only lets one GPU serve more people at once. ## Chain the three stages into an assistant Kiely (sec. 6.4.2) describes how most voice products work today. They cascade three models. ASR listens and produces text, an LLM reads the text and writes an answer, and TTS speaks the answer. A small voice activity detection model usually sits in front to notice when you start and stop talking. Single models that go speech to speech exist, but Kiely writes that at publication the open ones were not commercially viable, so the cascade is still the practical design. The cascade fits in surprisingly little memory. The file sizes below are approximate, and you should check the actual downloads. The whisper.cpp base English model is about 150 MB. A 1B parameter LLM at 4-bit quantization is about 0.8 GB. A medium Piper TTS voice is about 60 MB. That sums to about 1 GB of model weights for the whole loop, which is the basis for the 1 to 2 GB claim in the lead. ## Work the latency arithmetic for one turn The numbers below are made up to show the arithmetic. Measure your own in the exercise. Suppose ASR takes 1.0 second to transcribe your question, the LLM takes 0.3 seconds to read the prompt and then writes 20 tokens per second, the full answer is 60 tokens, and TTS needs 0.4 seconds to produce its first audio. - If you wait for the full answer before speaking, the decode takes 60 / 20 = 3.0 seconds, and the turn takes 1.0 + 0.3 + 3.0 + 0.4 = 4.7 seconds. - If TTS starts as soon as the first sentence exists, and that sentence is 15 tokens, the wait for text is 15 / 20 = 0.75 seconds, and the turn takes 1.0 + 0.3 + 0.75 + 0.4 = 2.45 seconds. Streaming the answer into TTS sentence by sentence cut the perceived latency almost in half, and no stage got faster. This is why Kiely prefers time to first sentence over time to first byte as the user-facing TTS metric. The user hears the first sentence while the LLM is still writing the rest. ## Try it You can build the loop in about 30 minutes with [whisper.cpp](https://yourwildcard.ai/docs/ecosystem/whisper-cpp.md), [llama.cpp](https://yourwildcard.ai/docs/ecosystem/llama-cpp.md), and Piper. No GPU is needed. All three run on a laptop CPU, and whisper.cpp and llama.cpp use the GPU on Apple silicon automatically. Install the pieces and fetch the models. The exact model files are listed on each project's page. ```bash git clone https://github.com/ggml-org/whisper.cpp && cd whisper.cpp cmake -B build && cmake --build build -j ./models/download-ggml-model.sh base.en brew install llama.cpp ffmpeg pip install piper-tts ``` Record a five second question, then run the three stages and time each one. On a Mac, `-f avfoundation -i ":0"` records from the default microphone. ```bash ffmpeg -f avfoundation -i ":0" -t 5 -ar 16000 question.wav time ./build/bin/whisper-cli -m models/ggml-base.en.bin -f question.wav -otxt -of question time llama-cli -hf ggml-org/Llama-3.2-1B-Instruct-GGUF -f question.txt -n 128 > answer.txt time (cat answer.txt | piper --model en_US-lessac-medium --output_file answer.wav) afplay answer.wav ``` Add the three times together for your end-to-end latency per turn, and note which stage took the longest. On most laptops it is the LLM decode. If you have a few more GB of memory free, swap the 1B model for [Bonsai 8B](https://huggingface.co/prism-ml/Bonsai-8B-gguf) and measure how the answer quality and the decode time both change. ## Check yourself 1. In one sentence each, what do ASR and TTS do? Expected answer: ASR converts spoken audio into text, and TTS converts text into spoken audio, so they are the same loop run in opposite directions. 2. What makes a modern TTS model an LLM rather than a separate kind of model? Expected answer: per Kiely (sec. 6.4), it is a fine-tuned LLM whose vocabulary was expanded with encoded audio tokens, plus a small audio decoder that turns the generated tokens into a waveform. 3. In the worked example, why did streaming the first sentence into TTS nearly halve the turn latency? Expected answer: the user starts hearing audio after 15 tokens of decode instead of 60, so the wait drops from 4.7 to 2.45 seconds even though every stage runs at the same speed. 4. Why is there no benefit to a TTS model decoding at 500 tokens per second for a single listener? Expected answer: per Kiely (sec. 6.4), 80 to 100 tokens per second already produces audio in real time, so extra speed only helps serve more concurrent streams. ## Next steps - [whisper.cpp](https://yourwildcard.ai/docs/ecosystem/whisper-cpp.md) covers the ASR stage in more depth. - [llama.cpp](https://yourwildcard.ai/docs/ecosystem/llama-cpp.md) covers running the LLM stage locally. - [Is speech-to-speech one model or a pipeline?](/blog/speech-to-speech-vs-asr-llm-tts-pipeline) compares this cascade against single-model voice systems. When you can do this, you can assemble a fully local three-stage voice loop and profile its per-stage latency. --- --- title: "Why does batch size trade latency for throughput in LLM serving?" description: "Bigger batches raise total tokens per second during memory-bound decode, but each user's stream slows. Continuous batching softens that penalty." audience: inference-engineer pillar: foundational-concept book: inference-engineering chapter_ref: "Ch. 7, sec. 7.2.1 (pp. 186-188)" status: published last_reviewed: 2026-07-07 canonical_url: https://yourwildcard.ai/blog/batch-size-latency-throughput-tradeoff --- # Why does batch size trade latency for throughput in LLM serving? Bigger batches raise total tokens per second by keeping the GPU busy during memory-bound decode, but each individual user's stream slows down. Continuous batching, also called in-flight batching, swaps requests at the token level to soften that penalty versus static batching. This is the cleanest framing of the serving frontier between latency and throughput. ## Understand why one batch shares one weight read Decode is memory bound. For every token it generates, the GPU reads the full set of model weights from memory and does one small multiply per request. The compute units finish their work long before the memory system finishes moving bytes, so at batch size 1 the GPU spends most of each step waiting on memory. Batching fills that idle compute. When eight requests sit in one batch, the GPU still reads the weights once per step, but that one read now produces eight tokens instead of one. The cost of the read is shared, so aggregate tokens per second goes up. The penalty lands on each user. Each decode step now carries eight requests' worth of compute and eight requests' worth of KV cache reads, so the step takes longer than it did at batch 1. Every user in the batch receives their next token a little later. Kiely's Inference Engineering (Ch. 7, sec. 7.2.1) states the rule directly: "Batch sizing trades off latency for throughput." Kiely's advice is to test several batch sizes against your own model, hardware, latency target, and budget, because no single batch size fits all four. ## Work through the ceiling arithmetic You can derive the shape of the tradeoff from memory bandwidth alone. These numbers are ceilings from arithmetic, not benchmarks. Take an 8 billion parameter model in FP16. The weights are about 16 GB. Every decode step reads all of them. On a GPU with 3.35 TB/s of memory bandwidth, the most decode steps you can run per second is 3350 divided by 16, which is about 209 steps per second. - At batch 1, each step yields 1 token. The ceiling is about 209 tokens per second in total, and the one user gets all 209. - At batch 4, each step yields 4 tokens. The ceiling is about 836 tokens per second in total. Each user's rate stays near 209 in this ideal model, because the weight read still dominates the step time. - At batch 8, the total ceiling is about 1,672 tokens per second, again with each user near 209. Real servers fall short of these ceilings, and the gap grows with batch size. Each added request adds compute and adds its own KV cache reads, so steps slow down and per-user tokens per second drops below the ideal number. The pattern to expect is that total throughput keeps climbing as you add requests, while per-user speed keeps falling. Your job is to find the batch size where total throughput is high and per-user latency is still inside your budget. That pair of numbers is your operating point. ## Pick a batching mode Kiely's Inference Engineering (Ch. 7, sec. 7.2.1) lists three ways a server can form batches. - **Static batching.** The server waits until the batch is full before it starts inference. Early requests sit in the queue while later ones trickle in, so early requests pay a long wait for nothing. - **Dynamic batching.** The server starts when the batch is full or when a timer runs out. The timer caps the queue wait, but a request that arrives just after a batch launches still waits for the next one. - **Continuous batching.** The server runs inference nonstop and swaps requests in as slots open up. Requests join and leave at the token level, so a new request does not wait for the current batch to finish. Kiely notes that engines like vLLM, SGLang, and TensorRT-LLM implement continuous batching, and that TensorRT-LLM calls it in-flight batching. Compared to static batching, it removes most of the queue time without giving up the shared weight read, which is why it is the default in modern serving stacks. Batch size also feeds your autoscaler. Kiely points out that the autoscaler's concurrency target and the replica's batch size should match. When every replica runs at its maximum concurrency, the system scales up, and when replicas keep launching half-full batches, it scales down. ## Try it Measure the tradeoff on your own machine with llama.cpp. This takes under 30 minutes with a model you already have. Start a server with 4 parallel slots. The `-np` flag sets how many requests decode at once, and the context is split across slots. ```bash llama-server -m ./model.gguf -c 16384 -np 4 --port 8080 ``` Write 20 short prompts, one per line, in `prompts.txt`. Then run them one at a time and record the total wall time. ```bash time while IFS= read -r p; do curl -s http://localhost:8080/v1/completions \ -H "Content-Type: application/json" \ -d "{\"prompt\": \"$p\", \"max_tokens\": 128}" > /dev/null done < prompts.txt ``` Now run the same 20 prompts with 4 in flight at once. ```bash time xargs -P 4 -I {} curl -s http://localhost:8080/v1/completions \ -H "Content-Type: application/json" \ -d '{"prompt": "{}", "max_tokens": 128}' \ -o /dev/null < prompts.txt ``` Compare the two total times to see the throughput gain. Then wrap each curl in its own `time`, or read the per-request timings from the server log, to see the per-request latency cost. Restart the server with `-np 8` and run the concurrent test again to get a third point on the curve. ## Check yourself 1. Why does batching raise throughput during decode? Expected answer: decode is memory bound, and one read of the weights now produces one token for every request in the batch, so the cost of moving bytes is shared. 2. What are your aggregate tokens per second and your per-user tokens per second at batch sizes 1, 4, and 8 on your own server? Expected answer: your three measured pairs from the exercise, with aggregate rising and per-user falling as the batch grows. 3. At which batch size did per-user latency cross your acceptable line? Expected answer: the specific size from your run, stated against a latency budget you wrote down before testing. 4. Why does continuous batching beat static batching on latency at the same batch size? Expected answer: requests join at the token level as slots open, so a new request does not wait in a queue for a full batch to form or finish. ## Next steps - Read the [batching guide](https://yourwildcard.ai/docs/technical-guides/batching.md) for how to set batch limits in a production server. - Read the [autoscaling guide](https://yourwildcard.ai/docs/technical-guides/autoscaling.md) to tie your chosen concurrency target to replica scaling. - Read [why is LLM prefill compute-bound but decode memory-bound?](/blog/prefill-compute-bound-decode-memory-bound) for the arithmetic behind the shared weight read. When you can do this, you can measure the batch-size frontier on a local server and choose an operating point against a stated latency budget. --- --- title: "How do you benchmark a local model when your budget is a power outlet, not a GPU cluster?" description: "When watts and RAM are the hard constraints, fix the machine, fix a power or thermal ceiling, and compare models on capability delivered within that envelope." audience: inference-engineer pillar: end-user-case-study book: transcript-theme chapter_ref: "Theme 1: Intelligence density (capability per GB/watt/dollar)" status: published last_reviewed: 2026-07-07 canonical_url: https://yourwildcard.ai/blog/benchmarking-on-a-power-outlet-budget --- # How do you benchmark a local model when your budget is a power outlet, not a GPU cluster? When watts and RAM are the hard constraints, benchmark inside the envelope. Fix the machine, fix a power or thermal ceiling, and compare models on the capability each one delivers within that envelope, not on what each could do with unlimited hardware. This framing routinely changes the model choice, because the leaderboard winner is often not the envelope winner. ## Fix the envelope before you run anything A benchmark on someone else's hardware answers someone else's question. If your deployment target is a mini PC in a clinic or a laptop on battery, the question is not "which model is most capable". The question is "which model delivers the most capability on this exact machine without tripping its limits". So the first step is to write the envelope down before you download a single model. An envelope has three parts. - The machine. One specific device with a known amount of RAM and a known chip. Every run in the benchmark happens on this device and no other. - The ceiling. A limit you refuse to cross, e.g., the model plus its working memory must stay under 12 GB of RAM, or the package power must stay under 30 watts sustained. Pick the limit that reflects your deployment, not the one that is easy to measure. - The workload. Your own eval prompts with a pass condition you decided in advance. The [benchmarking guide](https://yourwildcard.ai/docs/technical-guides/benchmarking.md) covers how to build this set. A model that produces better output but breaks the ceiling scores zero. That rule sounds harsh, but it matches reality. A model that swaps to disk or throttles the chip does not ship, so its offline quality is not available to your users. ## Compare models on density, not on absolute quality The theme this post draws on defines [intelligence density](https://yourwildcard.ai/docs/concepts/intelligence-density.md) as capability per GB, per watt, and per dollar. Inside a fixed envelope, density is the number that decides. The larger model almost always passes more evals in absolute terms. The envelope question is whether it passes enough more to justify what it takes from a machine that has little to give. The comparison has two stages. 1. Eliminate. Any model that breaks the ceiling is out, no matter its score. If the 8B model needs 9 GB and your ceiling is 6 GB of free RAM, the 8B model is not a worse choice. It is not a choice. 2. Rank. Among the models that fit, divide the eval pass rate by the resource each one consumes, e.g., passes per GB of resident memory, or passes per watt of sustained draw. The highest ratio wins the slot, because the resource it leaves free goes to the rest of the application or to a longer battery life. Note what this ordering does. It never asks where the models sit on a public leaderboard. A leaderboard compares models with the constraint removed, and your envelope is the constraint. The two rankings agree only by accident. ## Work one example with derived numbers Here is the arithmetic for a made-up but realistic case. Every number is an assumed input so you can follow the division, not a measured benchmark result. Rerun the math with your own measurements. The machine is a MacBook Air with 16 GB of RAM, and the ceiling is 8 GB free for the model. The workload is a 20-prompt extraction eval. - Model A is a 4B model in 4-bit GGUF. Its file is about 2.5 GB. Suppose it passes 14 of 20 prompts, a pass rate of 0.70. - Model B is an 8B model in 4-bit GGUF, e.g., [Bonsai 8B](https://huggingface.co/prism-ml/Bonsai-8B-gguf). Its file is about 5 GB. Suppose it passes 17 of 20, a pass rate of 0.85. Both fit under the 8 GB ceiling, so neither is eliminated and the choice goes to the ranking stage. Passes per GB for Model A is 0.70 divided by 2.5, which is 0.28. For Model B it is 0.85 divided by 5, which is 0.17. Model B wins on absolute quality and Model A wins on density, and the envelope decides which fact governs. If the app needs the extra 5.5 GB for a document cache, Model A wins. If the machine has nothing else to hold and the workload punishes the 3 extra failures, Model B earns its footprint. Now tighten the ceiling to 4 GB, e.g., the same app must run on the 8 GB machines in the fleet. Model B no longer fits, so the elimination stage removes it before quality is even discussed. The model choice changed and no model got better or worse. Only the envelope moved, and that is the point of writing it down first. Say which constraint eliminated a model, not just that it lost. "The 8B model broke the RAM ceiling" and "the 8B model fit but ranked lower per GB" lead to different decisions when the fleet gets new hardware next year. ## Try it Benchmark two models of different sizes inside one envelope. Budget 30 minutes. No GPU is required, and any recent laptop with 16 GB of RAM works. Write 10 prompts for one task into `prompts.txt`, one per line, and decide the pass condition before you run. ```bash # Install llama.cpp (macOS) brew install llama.cpp # Run each model over the prompts while logging peak memory: while IFS= read -r p; do /usr/bin/time -l llama-cli -m model-4b-q4_k_m.gguf \ -p "$p" -n 256 --temp 0 -no-cnv 2>> mem-4b.log done < prompts.txt while IFS= read -r p; do /usr/bin/time -l llama-cli -m model-8b-q4_k_m.gguf \ -p "$p" -n 256 --temp 0 -no-cnv 2>> mem-8b.log done < prompts.txt # Peak resident memory per run is the "maximum resident set size" line: grep "maximum resident set size" mem-4b.log mem-8b.log # If you can, log power draw in a second terminal during the runs (macOS): sudo powermetrics --samplers cpu_power -i 1000 -n 30 ``` On Linux, use `/usr/bin/time -v` for peak memory and check whether your machine exposes power through `powerstat` or a similar tool. If you cannot read power, RAM alone is enough for the exercise. Score the passes for each model, divide each pass rate by its peak memory in GB, and rank the two models by that ratio. Then rerun the whole eval once and check that the ranking holds. ## Check yourself 1. Is your ranking stable across the re-run? If the order flipped, your prompts are too few or your pass condition is too vague, so tighten the condition before you trust either ranking. 2. Which constraint would eliminate the larger model on your target machine, RAM, thermals, or speed? You should be able to name one and point at the log line that shows it, e.g., the peak resident memory that crossed your ceiling. 3. Did the denser model and the higher-scoring model differ? If they did, say in one sentence which one ships and why the envelope makes that the right call. ## Next steps - [Intelligence density](https://yourwildcard.ai/docs/concepts/intelligence-density.md) defines capability per GB, per watt, and per dollar, which is the ratio this whole method ranks on. - [Benchmarking](https://yourwildcard.ai/docs/technical-guides/benchmarking.md) covers building the eval set and reporting the results so a second person can rerun them. - [What does 'intelligence density' actually measure?](/blog/what-is-intelligence-density) works the same ratio for a model selection argument rather than a benchmark design. When you can do this, you can run a benchmark inside a fixed hardware envelope and select a model on the capability it delivers within your budget. --- --- title: "What's the difference between benchmarking and profiling inference?" description: "Benchmarking tells you how the system performs. Profiling tells you why, down to per-kernel time, and you need it only when you write engine code." audience: researcher pillar: ecosystem-player book: inference-engineering chapter_ref: "Ch. 4, sec. 4.5.3 Profiling Performance" status: published last_reviewed: 2026-07-07 canonical_url: https://yourwildcard.ai/blog/benchmarking-vs-profiling-inference --- # What's the difference between benchmarking and profiling inference? Benchmarking tells you how the system performs. Profiling, with tools like PyTorch Profiler and Nsight, tells you why, down to per-kernel time. You need profiling only when you write or contribute engine code. For a workflow of configuring and benchmarking, it adds nothing. Knowing when to skip it saves days, and knowing when it is required is what finds fused kernel opportunities. ## Separate the two questions Kiely's Inference Engineering (Ch. 4, sec. 4.5.3) draws the line in one sentence: "Benchmarking tells you how your system is performing; profiling tells you why". A benchmark returns a single figure, e.g., a P90 TTFT of 350 ms. P90 TTFT is the time to first token that 90 percent of requests beat. A profiler shows where each of those milliseconds went inside the inference process, operation by operation and kernel by kernel. A kernel is one GPU program, e.g., a matrix multiply or an attention step. So before you open any tool, name your question. - "Is batch size 16 faster than batch size 8 for my traffic?" is a benchmarking question. You compare two whole-system figures. - "Why does the forward pass spend 40 percent of its GPU time outside the matrix multiplies?" is a profiling question. You need per-operation timing to answer it. The tools do not overlap. A benchmark cannot tell you which kernel is slow, and a profiler run under fake single-request load cannot tell you what production latency will be. ## Decide whether you need profiling at all The book is direct about this. Most inference engineers never need profiling in their daily work. If you serve models through an already fast engine, e.g., TensorRT-LLM or vLLM, your job is a cycle of changing configuration and benchmarking the result. Kiely calls profiling in that workflow extraneous. The engine authors already did the kernel work, and no profiler trace will change which config flag you flip next. Profiling becomes required in three situations the book names. - You contribute to an inference framework such as vLLM or SGLang. - You write your own inference service in PyTorch. - You work on a new modality, e.g., video generation, where fast engines do not exist yet. In those cases the profiler is what finds the expensive step. The book's worked case is the fused kernel story. A PyTorch Profiler trace shows activation functions taking too long because of excess memory reads, so you write a fused kernel that runs the activations alongside attention, insert it, and then rerun the system-level benchmark to see if the latency target holds. Note the order. Profiling finds the target, the fix is code, and benchmarking judges the fix. For a researcher this is the routing rule. If your fix will be a config change, benchmark. If your fix will be code inside the engine, profile first. ## Read a profile as a budget Here is a made-up but arithmetically honest example of what a profiler gives you that a benchmark cannot. Suppose your benchmark says one decode step takes 40 ms of GPU time. That figure is the whole answer a benchmark can give. A profiler splits it into a table of operations, and the split is where the work is. | Op (by GPU time) | GPU time | Share of 40 ms | |---|---|---| | Matrix multiplies (linear layers) | 22 ms | 55% | | Attention kernels | 10 ms | 25% | | Activations and elementwise ops | 6 ms | 15% | | Everything else | 2 ms | 5% | These numbers are invented for the arithmetic, not measured. The reading is what transfers. The activations line should be small because those ops do little math, so 6 ms out of 40 ms, which is 15 percent, is a flag. If a fused kernel removed even two thirds of it, you would save 4 ms, which is a 10 percent step-time cut. That estimate is what the profiler buys you, because it turns "make it faster" into "these 6 ms are the cheapest 6 ms to attack". Then you benchmark the fix, because only the whole-system number tells you whether the saving survived batching and real traffic. The tool tiers go deeper as you go down. PyTorch Profiler gives per-op CPU time, GPU time, and memory, and it is the easy first stop. NVIDIA Nsight Systems traces the whole system across GPUs and their links. NVIDIA Nsight Compute analyzes a single CUDA kernel's compute and memory behavior. Start at the top and only go down when the question forces you. ## Try it Time budget is about 20 minutes. PyTorch Profiler runs on CPU anywhere, so you can do this on a laptop. The GPU time column only appears if you have a CUDA GPU, so on a Mac or CPU-only machine read the CPU time column instead and the routing lesson is the same. Any small model works, e.g., a small Hugging Face checkpoint, and if you want a public small model to poke at, [Bonsai 8B](https://huggingface.co/prism-ml/Bonsai-8B-gguf) is on the Hub, though for this exercise a model that loads in plain PyTorch is simpler. 1. Install PyTorch, then write a forward pass of about 10 lines: ```python import torch from torch.profiler import profile, ProfilerActivity from transformers import AutoModelForCausalLM, AutoTokenizer tok = AutoTokenizer.from_pretrained("gpt2") model = AutoModelForCausalLM.from_pretrained("gpt2").eval() ids = tok("Profiling finds the expensive step.", return_tensors="pt").input_ids acts = [ProfilerActivity.CPU] + ([ProfilerActivity.CUDA] if torch.cuda.is_available() else []) with profile(activities=acts) as prof, torch.no_grad(): model(ids) print(prof.key_averages().table(sort_by="self_cpu_time_total", row_limit=10)) ``` 2. On a CUDA machine, move the model and ids to `cuda` and sort by `self_cuda_time_total` instead. 3. Read the table and write down the top three ops by GPU time (or CPU time), each with its share of the total. 4. Say in one sentence what you would attack first if you had to cut step time by 10 percent. ## Check yourself - **A teammate asks whether your server holds its latency target at double the traffic. Benchmark or profile?** Expected answer: benchmark. The question is about a whole-system figure under load, and no per-kernel data changes the answer. - **Your own PyTorch serving code is slower than the same model in vLLM. Benchmark or profile?** Expected answer: profile your code. You wrote the engine code, so the fix will be code, and you need per-op time to find where yours loses. - **You run vLLM with default flags and want faster TTFT. Should you reach for Nsight?** Expected answer: no. Kiely's rule is that with a high-performance engine your workflow is configuration and benchmarking, and profiling is extraneous unless you plan to contribute kernel-level changes. - **In the worked table, why does the 6 ms activations line get attacked before the 22 ms matrix multiply line?** Expected answer: the matrix multiplies are doing the model's math and are already the target of years of kernel work, while the activations line is large relative to the little math it does, so it is the likelier sign of waste, e.g., excess memory reads a fused kernel can remove. ## Next steps - [Benchmarking guide](https://yourwildcard.ai/docs/technical-guides/benchmarking.md) covers how to get the whole-system numbers this post routes to. - [Observability guide](https://yourwildcard.ai/docs/technical-guides/observability.md) covers watching those numbers in production rather than in a test. - [How do I benchmark an LLM server properly?](/blog/how-to-benchmark-an-llm-server) is the companion post for the benchmarking side, including noise bounds and one variable per comparison. When you can do this, you can route a performance investigation to benchmarking or profiling and extract a top-ops table from a profiler run. --- --- title: 'Binary versus Ternary Bonsai 27B' description: "Choose between PrismML's 1 bit and ternary Bonsai 27B models based on memory, quality, speed, and device limits." audience: local-ai-builder pillar: bonsai-27b status: published last_reviewed: 2026-07-15 canonical_url: https://yourwildcard.ai/blog/binary-vs-ternary-bonsai-27b --- # Binary versus Ternary Bonsai 27B Use Ternary Bonsai 27B when quality is the main goal and a laptop has enough memory. Use Binary Bonsai 27B when the smallest footprint is required, especially on a phone. PrismML reports 80.49 for ternary and 76.11 for binary on its 15 benchmark thinking average. ## The main tradeoff - Ternary uses minus one, zero, and plus one. Its ideal language model size is about 5.9 GB. - Binary uses minus one and plus one. Its deployed language model size is about 3.9 GB. - Ternary retains 94.6 percent of PrismML's FP16 benchmark average. - Binary retains 89.5 percent of the same benchmark average. ## Choose for the device PrismML positions ternary as the laptop quality option and binary as the phone footprint option. The ternary model is too large for the per app memory budget PrismML cites for the iPhone 17 Pro Max. The binary model leaves more room for context and runtime buffers. ## Questions people ask ### Is ternary always faster? No. Binary moves fewer weight bytes and can decode faster when memory bandwidth is the limit. Runtime and kernel support also affect speed. ### Is binary good enough for tool use? PrismML reports lower agentic and tool calling scores for binary than ternary. Test your own tools and failure cases before choosing it for an automated workflow. ## Sources - [PrismML Bonsai 27B announcement](https://prismml.com/news/bonsai-27b) - [PrismML Bonsai 27B whitepaper](https://github.com/PrismML-Eng/Bonsai-demo/blob/main/bonsai-27b-whitepaper.pdf) - [PrismML Bonsai 27B documentation](https://docs.prismml.com/models/bonsai-27b) - [PrismML Bonsai 27B collection on Hugging Face](https://huggingface.co/collections/prism-ml/bonsai-27b) ## Related Bonsai 27B lessons - [How good is Ternary Bonsai 27B?](/blog/ternary-bonsai-27b-quality-benchmarks) - [How large is Ternary Bonsai 27B?](/blog/ternary-bonsai-27b-model-size) - [Ternary Bonsai 27B limitations and verification checklist](/blog/ternary-bonsai-27b-limitations) The benchmark numbers on this page describe one checkpoint, runtime, machine, and test shape. Reproduce the test on the hardware and workload you plan to use before making a product decision. --- --- title: 'What the ecosystem is saying about Bonsai 27B' description: "A link first record of The Information's report, PrismML's launch thread, Khosla team amplification, and public reactions to Bonsai 27B." audience: local-ai-builder pillar: bonsai-27b status: published last_reviewed: 2026-07-15 canonical_url: https://yourwildcard.ai/blog/bonsai-27b-ecosystem-reactions --- # What the ecosystem is saying about Bonsai 27B The main public story has two parts. The Information reported that Apple had talked with PrismML about possible uses of its compression work. PrismML then released the 27B checkpoints, benchmarks, and demos. Public reactions focused on the phone size claim, local agents, and the need to test quality outside vendor benchmarks. ## The Information is the centerpiece Aaron Tilley reported on July 9 that PrismML had run a compressed Qwen3.6 27B model on an iPhone 17 Pro and that Apple had discussed possible uses of the technology with the company. The article is behind a subscription wall, so this page links to it and does not copy its text. PrismML released Bonsai 27B on July 14. Its own announcement gives the model sizes, benchmark claims, formats, and license. These are company claims until independent tests reproduce them. - [The Information report by Aaron Tilley](https://www.theinformation.com/articles/khosla-backed-startup-claims-breakthrough-largest-ever-ai-model-iphone), July 9, 2026 - [PrismML launch announcement](https://prismml.com/news/bonsai-27b), July 14, 2026 - [CNBC follow-up by MacKenzie Sigalos](https://www.cnbc.com/2026/07/14/apple-prismml-ai-compression-iphone.html), July 14, 2026 ## PrismML's public launch thread PrismML used six posts to separate the launch, benchmark, phone memory, agent demo, intelligence density, and local agent arguments. The links below go to the original posts. - [Launch and model overview](https://x.com/prismml/status/2077084891284721827) - [Why local agent loops are part of the pitch](https://x.com/prismml/status/2077084897819381958) - [Intelligence density claim](https://x.com/prismml/status/2077084893574836475) - [Phone memory threshold](https://x.com/prismml/status/2077084901988532427) - [Benchmark retention summary](https://x.com/prismml/status/2077084895789359468) - [Hermes agent demo on an RTX 5090](https://x.com/prismml/status/2077084899904024918) ## What the Khosla team amplified Kanu Gulati is a Khosla Ventures partner and lists PrismML among the companies he led. His public feed reshared the reactions from Ion Stoica, Together AI, NVIDIA RTX Spark, and other launch participants. That shows investor support and distribution. It is not an independent benchmark. - [Kanu Gulati's public X profile](https://x.com/KanuGulati) - [Ion Stoica called it an impressive systems engineering result](https://x.com/istoica05/status/2077162153241641140) - [Together AI announced hosted access](https://x.com/togethercompute/status/2077115824825860366) - [NVIDIA RTX Spark highlighted local GPU support](https://x.com/NVIDIARTXSpark/status/2077111337512239353) ## Independent reactions and open questions Vaibhav Sisinty summarized the size and benchmark claims for a broad builder audience. Other early posts focused on whether the release changes local agent economics. The strongest open question is still task level quality, especially for tool use and long agent loops. Our local work answers a narrower question. The public ternary MLX checkpoint loads on a 24 GB M4 Pro Mac, generates at about 21 to 23 tokens per second in controlled 4K runs, and can return a structured tool call. It does not verify the iPhone result or the full vendor benchmark suite. - [Vaibhav Sisinty's Bonsai 27B summary](https://x.com/VaibhavSisinty/status/2077132269865877910) - [9to5Mac separates the model release from the Apple discussion](https://9to5mac.com/2026/07/14/prismml-releases-bonsai-27b-claiming-first-major-ai-model-of-its-size-fit-for-iphone/) ## Questions people ask ### Did Apple confirm a deal with PrismML? No public deal has been confirmed. Reports describe early evaluation and discussions. ### Did Khosla Ventures publish an independent benchmark? No public independent benchmark was found. The public record shows investment support and amplification of the launch. ### Are all X reactions independent? No. PrismML's posts are company statements, and investor or platform partner posts have their own interests. This page labels each group. ## Sources - [The Information report by Aaron Tilley](https://www.theinformation.com/articles/khosla-backed-startup-claims-breakthrough-largest-ever-ai-model-iphone) - [PrismML Bonsai 27B announcement](https://prismml.com/news/bonsai-27b) - [CNBC report on Bonsai 27B and Apple evaluation](https://www.cnbc.com/2026/07/14/apple-prismml-ai-compression-iphone.html) - [PrismML Bonsai 27B launch post on X](https://x.com/prismml/status/2077084891284721827) - [PrismML Hermes agent demo on X](https://x.com/prismml/status/2077084899904024918) - [Together AI launch reaction on X](https://x.com/togethercompute/status/2077115824825860366) - [NVIDIA RTX Spark launch reaction on X](https://x.com/NVIDIARTXSpark/status/2077111337512239353) - [Ion Stoica launch reaction on X](https://x.com/istoica05/status/2077162153241641140) - [Vaibhav Sisinty Bonsai 27B summary on X](https://x.com/VaibhavSisinty/status/2077132269865877910) - [Kanu Gulati of Khosla Ventures on X](https://x.com/KanuGulati) ## Related Bonsai 27B lessons - [How good is Ternary Bonsai 27B?](/blog/ternary-bonsai-27b-quality-benchmarks) - [How to compare PrismML benchmarks with local Bonsai 27B tests](/blog/ternary-bonsai-27b-vendor-vs-local-benchmarks) - [Ternary Bonsai 27B passed a structured tool call](/blog/ternary-bonsai-27b-tool-calling-test) The benchmark numbers on this page describe one checkpoint, runtime, machine, and test shape. Reproduce the test on the hardware and workload you plan to use before making a product decision. --- --- title: "Does the 1-bit Bonsai whitepaper prove the efficiency thesis this wiki is built on?" description: "The whitepaper is a working example of the argument that memory movement and inference cost, more than training or raw compute, decide where a model can run. Here is what it shows, and what stays unverified." audience: ml-product-team pillar: ecosystem-player book: transcript-theme chapter_ref: "Orientation: the Stoica and Khosla lenses" status: published last_reviewed: 2026-07-07 canonical_url: https://yourwildcard.ai/blog/bonsai-whitepaper-and-the-efficiency-thesis --- # Does the 1-bit Bonsai whitepaper prove the efficiency thesis this wiki is built on? This wiki is organized around two arguments about where AI is heading. The 1-bit Bonsai 8B whitepaper is a concrete test of both. The paper takes an 8B model, stores every weight in about one bit, and reports that the result runs 5 to 8 times faster on the same hardware while keeping most of its benchmark scores. This post explains which parts of the whitepaper support the argument, which numbers are measured versus estimated, and which related claims about the training side are still only social posts. ## State the argument first The [orientation page](https://yourwildcard.ai/docs/start-here/orientation.md) sets out the two lenses this wiki reads the ecosystem through. Both point at the same shift. Ion Stoica works on open source AI systems, distributed compute, serving, and evaluation. He splits the stack into three parts, which are data, training, and serving ([Columbia Engineering interview](https://www.engineering.columbia.edu/about/news/ion-stoica-highlights-open-source-efficiency-across-ai-stack)). The systems he is known for, Ray and vLLM, are about making the serving part reliable and cheap across many kinds of hardware. The [Ray page](https://yourwildcard.ai/docs/ecosystem/ray.md) covers how those pieces fit together. Vinod Khosla argues that intelligence should get cheaper and more common, and less limited by energy and datacenter capacity. Because of that, this wiki tracks capability per dollar, per watt, per gigabyte, and per device rather than parameter count. Put together, the two lenses make one claim. The hard problem in AI is no longer only training a capable model. It is serving that model at a price, a power draw, and a size that let it run where people actually need it, which includes phones, laptops, and edge devices, not only cloud GPUs. Training is a cost you pay once or a few times. Serving is a cost you pay on every request, so at scale it is the cost that dominates. ## Read what the whitepaper measures The whitepaper is [1-bit Bonsai 8B: End-to-end 1-bit language model deployment](https://yourwildcard.ai/whitepaper/1-bit-bonsai-8b-whitepaper.pdf) (PrismML, March 31, 2026). The [whitepaper benchmarks page](https://yourwildcard.ai/docs/prismml/whitepaper-benchmarks.md) is the plain tour of every number, and this section pulls out the parts that speak to the argument above. The model is Qwen3-8B with its architecture unchanged. The new part is the storage format. Each weight is kept as one sign bit, plus one shared 16-bit scale for every group of 128 weights, which works out to a little over one bit per weight. The kernel reads the bits and the scale during the matrix multiply, so no full precision copy of the weights is ever built in memory. The size drop is the first result. The full precision model is 16.38 GB on disk. The 1-bit build is about 1.15 GB in the whitepaper, which the current [Hugging Face file listing](https://huggingface.co/prism-ml/Bonsai-8B-gguf/tree/main) shows as 1.16 GB. That is roughly a 14 times reduction. The speed and energy results follow from the smaller size. On an RTX 4090, a Mac M4 Pro, and other machines, generating tokens runs 5.4 to 8.4 times faster than the full precision build of the same model on the same machine. Energy per generated token drops 4 to 6 times on the Mac and the desktop GPU. On an iPhone 17 Pro Max the model generates about 44 tokens per second, but the paper is explicit that the phone energy figure is estimated from a profiler and battery drain, not metered, so quote it with that caveat. The quality results are where the tradeoff shows. Across six benchmarks covering knowledge, reasoning, math, coding, instruction following, and tool calling, the 1-bit model averages 70.5. Its own full precision base averages 79.3, so the 1-bit build gives up points, most visibly on knowledge and tool calling. What it buys with those points is a model one fourteenth the size that still matches or beats several models ten times its size on the average. The paper's own summary metric is intelligence density, which is capability divided by size in gigabytes, defined so that gains near the top of the scale count for more. The three Bonsai models lead every model in the 1.2B to 9B comparison on this measure. The 8B build scores 1.060 per GB, the next best model scores 0.549, and the full precision Qwen3-8B scores 0.096. The [intelligence density post](/blog/what-is-intelligence-density) explains why this ratio, and not the raw average, is the number to compare. ## See why the speed comes from movement, not math This is the part that ties the whitepaper to the argument. One-bit weights do not make the arithmetic easier. In fact the kernel does extra work to unpack each bit during the multiply. The model still runs faster because when a model generates one token at a time, the slow step is reading the weights out of memory, not doing the multiplies. A 14 times smaller weight file means 14 times fewer bytes to read for each token. The [bandwidth ledger guide](https://yourwildcard.ai/docs/technical-guides/bandwidth-ledger.md) walks through this arithmetic in full. This is the exact claim behind the two lenses above. The bottleneck people are actually paying for is data movement and memory bandwidth, so a change that cuts the bytes moved per token buys speed, energy, and a smaller footprint at once, without touching the arithmetic. That is why the whitepaper reads as a test of the argument rather than one more quantization result. It attacks the movement cost at the source, measures the effect on real hardware, and reports the quality it gives up in the same tables, so a reader can see the trade instead of guessing at it. ## Fit the training side, and mark what is unconfirmed The argument has two halves, which are training a model and serving it. The whitepaper covers the serving half. The training half in the same story is EasyDeL, an open source framework built on JAX for training, fine tuning, and serving large models ([EasyDeL on GitHub](https://github.com/erfanzar/EasyDeL)). Its latest stable release is v0.2.0.2, from January 2026, and its repository documents an inference engine called eSurge with continuous batching and a paged KV cache. Both of those facts come from the repository itself, so they are safe to state. The rest of the EasyDeL story that circulates with the Bonsai whitepaper comes only from posts on X, so under the [source policy](https://yourwildcard.ai/docs/sources/source-policy.md) it stays a lead, not a fact. A developer preview described a development branch called vnext, a v0.4.0 release that had not landed yet, new components named SpectraX, eRay, and ejKernel, more parallelism and reinforcement learning trainers, and a speedup of up to 32 percent over the main branch. A PrismML co founder amplified that preview. None of that is confirmed against a release, a changelog, or a benchmark log, and the claim that EasyDeL and PrismML are the same team also comes only from those posts. The [X discovery notes](https://yourwildcard.ai/docs/sources/x-discovery-notes.md) hold these signals in the queue with the sources that could settle them. So the honest version of the two sided story is this. The serving half is measured and published in a whitepaper you can read and reproduce. The training half has a real, verifiable open source project attached to it, and a set of newer claims about that project that are still waiting for a primary source. Keep the two apart when you present them. When you use this whitepaper to argue that efficiency is the main story in AI, cite the paper for the serving numbers and the model card for the file size. Cite the EasyDeL repository only for the stable release and eSurge. Do not repeat the vnext, SpectraX, or 32 percent claims as fact until they have an official source, because right now they are enthusiasm on X and nothing more. ## Try it You can check the movement claim yourself in about 20 minutes on a laptop or a desktop GPU. No second model download is needed. The goal is to see that the generation speed is set by how fast the weights stream through memory. 1. Download the 1-bit Bonsai 8B model and run it with the [llama.cpp recipe](https://yourwildcard.ai/docs/build-and-run/bonsai-llamacpp.md): ```bash pip install -U "huggingface_hub[cli]" hf download prism-ml/Bonsai-8B-gguf --local-dir ./bonsai ls -lh ./bonsai/*.gguf # note the file size in GB, call it S ``` 2. Generate some tokens and read the tokens per second the tool prints at the end, and call that number R: ```bash llama-cli -m ./bonsai/.gguf \ -p "Explain what memory bandwidth is in two sentences." \ -n 128 --temp 0 -no-cnv 2>&1 | tee run.log ``` 3. Multiply S by R. That product is roughly how many gigabytes of weights the machine reads every second while generating, because the model reads the whole weight file once per token. 4. Look up the memory bandwidth of your chip in gigabytes per second, then compare it to your S times R number. For a rough check, the paper reports about 368 tokens per second for this model on an RTX 4090. With a 1.15 GB file that is about 423 gigabytes per second of weight reads, against roughly 1000 gigabytes per second of bandwidth on that card. The reads take up a large part of the available bandwidth, which is why generation is limited by movement and not by the math. ## Check yourself - **Why does storing weights in one bit speed up generation even though the kernel does extra work to unpack each bit?** Expected answer: generation one token at a time is limited by reading the weights out of memory, and a smaller file means fewer bytes read per token, so the saved reading time is larger than the added unpacking time. - **A colleague says the whitepaper proves 1-bit models are as good as full precision. What is the honest correction?** Expected answer: the 1-bit model averages 70.5 against 79.3 for its full precision base, so it gives up points, most visibly on knowledge and tool calling. It trades some quality for about a 14 times smaller size, and the tables show exactly where. - **Which claims in the two sided story can you publish today, and which cannot?** Expected answer: the whitepaper serving numbers and the EasyDeL stable release and eSurge engine can be published with their sources. The EasyDeL vnext, SpectraX, and 32 percent claims cannot, because they come only from X posts and have no official source yet. ## Next steps - [Whitepaper benchmarks](https://yourwildcard.ai/docs/prismml/whitepaper-benchmarks.md) gives the full table for every number summarized here. - [The bandwidth ledger](https://yourwildcard.ai/docs/technical-guides/bandwidth-ledger.md) works through why decode speed is set by memory movement. - [What does intelligence density measure?](/blog/what-is-intelligence-density) explains the capability per gigabyte ratio the paper leads with. - [Ray and Anyscale](https://yourwildcard.ai/docs/ecosystem/ray.md) covers the serving systems behind the argument in this post. - [X discovery notes](https://yourwildcard.ai/docs/sources/x-discovery-notes.md) is where the unverified EasyDeL signals live until someone confirms them. When you can do this, you can explain the whitepaper as a test of the efficiency argument, separate its measured results from its estimated and unverified ones, and defend the split with numbers. --- --- title: "Can 1-bit models do tool calling?" description: "Yes, but not unconstrained. An independent benchmark found Bonsai 8B 1-bit scored 0% raw and 92% with grammar-constrained decoding. The capability is in the weights; the reliability comes from where you put the constraint." audience: inference-engineer pillar: foundational-concept book: transcript-theme chapter_ref: "Community benchmark (r/LocalLLaMA, 2026-07) + our docs chat design notes" status: published last_reviewed: 2026-07-07 canonical_url: https://yourwildcard.ai/blog/can-1-bit-models-do-tool-calling --- # Can 1-bit models do tool calling? Yes, but not unconstrained. An independent community benchmark ran Bonsai 8B 1-bit on 30 CPU tool-calling cases: 0% valid tool calls raw, 92% with grammar-constrained decoding, the best score in the test from a 1.16 GB file half the size of its nearest rival. The semantic capability is in the weights. The reliability comes from where you put the constraint. The numbers come from a single Reddit benchmark: 30 deterministic cases, temperature 0, one run, the author's own harness, posted with the author's own caveat that it is "a signal, not a leaderboard." By this site's [source policy](https://yourwildcard.ai/docs/sources/source-policy.md) that is community evidence, not a published benchmark. We cite it because it is, as far as we can tell, the first independent tool-calling eval of a 1-bit Bonsai model, and because its shape matches what we found building the [docs chat](https://yourwildcard.ai/docs/build-and-run/docs-chat.md). ## Read what the benchmark actually found The test compared six small models on tool-call generation with llama.cpp on CPU, each run twice: raw, and with a GBNF grammar that constrains decoding to valid tool-call JSON. | Model (quant, size) | Raw | With grammar | | --- | --- | --- | | Bonsai 8B Q1_0 (1.16 GB) | 0% | 92% | | Granite 4.1 3B Q4_K_M (2.0 GB) | 72% | 88% | | Qwen2.5-Coder 7B | 68% | 84% | | Qwen2.5-Coder 3B | 0% | 84% | | Qwen3 8B | 0% | 84% | | BitNet b1.58 2B | 0% | 44% | Two things stand out. First, raw format discipline does not track capability: three capable models scored 0% raw, and the grammar rescued all of them except BitNet. Second, once format is externally guaranteed, the 1-bit Bonsai outscored everything, including models with twice its footprint. The grammar did not add intelligence. It removed a failure mode so the intelligence could show. ## Understand why aggressive quantization breaks format first A tool call fails if one token is wrong: a missing brace, a stray word before the JSON, a hallucinated argument name. Generation samples one token at a time, so producing 200 structurally perfect tokens in a row is a chain of independent chances to slip. Quantization noise raises the per-token slip rate slightly, and the chain multiplies it. Semantic quality degrades gracefully with quantization; format validity degrades catastrophically, because format is the one task where a single token ruins everything. Grammar-constrained decoding fixes this at the sampler. At every step, the runtime masks out every token that would violate a formal grammar, so the model can only choose among continuations that keep the output valid. The model still picks which tool and which arguments; the grammar guarantees the envelope. llama.cpp ships this as [GBNF grammars](https://github.com/ggml-org/llama.cpp/blob/master/grammars/README.md); server APIs expose the same idea as structured outputs or JSON mode. ## Compare it with how our docs chat is constrained Our [Ask the docs chat](/chat) runs 1-bit Bonsai 1.7B in the browser over WebGPU, and it uses no grammar at all — not by preference, but because grammar-constrained decoding is not available in transformers.js today. That forced us to put the constraint in the harness instead, and the design generalizes: - **Constrain the input, not just the output.** Retrieval picks two pages and extracts only the paragraphs that match the question, so the model reasons over a few thousand characters of curated context instead of an open field. - **Shape the prompt for the failure mode.** Our eval showed the model missed answers when the question came after the excerpts. Question-first ordering fixed it. That is a constraint on attention, applied with zero tokens of overhead. - **Demand abstention.** The system prompt makes "The docs do not cover that yet" the required output when context is missing. In our gold-question runs the model never invented a number; its failure mode was abstaining too eagerly, which is the failure mode you want. - **Verify outside the model.** Every answer links its source pages, so the reader can check the claim in one click. The UI is part of the constraint system. The tasks differ, and that is the point. Tool calling needs a formally valid artifact, so the constraint belongs at the sampler. Grounded question answering needs faithful prose, so the constraint belongs in the context, the prompt, and the verification loop. Same 1-bit model family, different constraint placement, both usable. ## Pick your constraint by your task | Your task | Where the constraint goes | Tooling | | --- | --- | --- | | Tool calling, agents, structured extraction | Sampler: grammar or structured outputs | llama.cpp GBNF, server JSON mode, outlines-style libraries | | Grounded Q&A over documents | Harness: retrieval, prompt order, abstention rule, source links | Any runtime; see [our design notes](https://yourwildcard.ai/docs/learning-paths/designing-the-docs-chat.md) | | Browser or WebGPU deployment | Harness only, until grammar support lands in web runtimes | transformers.js; watch for structured-output support upstream | | Anything user-facing | Both, plus an eval with gold cases before you ship | 20 to 30 deterministic cases at temperature 0 is enough to catch a 0%-raw surprise | The commenter on the benchmark asked whether this scales to bigger models and to mixed reasoning-plus-tool-use workloads. The honest answer is that nobody has published that yet for 1-bit models; the [contributor guide](https://yourwildcard.ai/docs/contribute/contributor-guide.md) explains how to submit a run if you measure it first. ## Try it Reproduce the shape of the finding on your machine in about 20 minutes with any GGUF model: ```bash # 1. A tool-call prompt, raw llama-cli -m Bonsai-8B-Q1_0.gguf --single-turn -n 200 \ -p 'Call get_weather with city="Tokyo". Reply with JSON only: {"tool": ..., "args": {...}}' # 2. The same prompt, grammar-constrained to JSON llama-cli -m Bonsai-8B-Q1_0.gguf --single-turn -n 200 \ --grammar-file grammars/json.gbnf \ -p 'Call get_weather with city="Tokyo". Reply with JSON only: {"tool": ..., "args": {...}}' ``` Run each five times. Count how many raw outputs parse as JSON versus grammar-constrained ones. The gap you see is the benchmark's whole story in miniature. ## Check yourself 1. Why does quantization hurt format validity more than answer quality? (Expected: format fails on any single bad token, so per-token error compounds across the sequence; semantic quality averages out.) 2. Does a GBNF grammar make a model smarter? (Expected: no. It masks invalid tokens so existing capability is not disqualified by formatting slips.) 3. Why does our browser chat not use a grammar? (Expected: transformers.js has no grammar-constrained decoding yet, so the constraint moved to retrieval, prompt order, abstention, and source links.) ## Next steps - [How the docs chat was designed](https://yourwildcard.ai/docs/learning-paths/designing-the-docs-chat.md), the harness-side constraint story in full. - [Verify quantized model quality](/blog/verify-quantized-model-quality), the eval discipline that catches a 0%-raw surprise before your users do. - [What do TTFT and tokens per second measure?](/blog/what-do-ttft-and-tps-measure) if you are benchmarking the same models for speed. When you can do this, you can decide where a constraint belongs for any small-model deployment and defend the choice with a 30-case eval instead of a vibe. --- --- title: "Do I need an LLM for ticket routing, or is a classifier enough?" description: "For a bounded label set, a fine-tuned 100 MB classifier usually beats a prompted LLM on accuracy, cost, and latency. Run the bake-off before defaulting to an LLM." audience: ml-product-team pillar: end-user-case-study book: building-ai-products chapter_ref: "Nika, Ch. 1, 'The Stages of AI Evolution'; 'Traditional AI' subsection" status: published last_reviewed: 2026-07-07 canonical_url: https://yourwildcard.ai/blog/classifier-or-llm-for-ticket-routing --- # Do I need an LLM for ticket routing, or is a classifier enough? Usually a classifier is enough. For a bounded label set, a fine-tuned 100 MB classifier tends to beat a prompted LLM on the three axes that decide production fitness: accuracy on your labels, cost per request, and P95 latency. The LLM wins when labels shift weekly or when inputs need reasoning. Run the comparison before defaulting to generation, because it is an afternoon of work. ## Name the task before you pick the tool Ticket routing is a fixed decision. A ticket comes in, and the system must assign it one label from a list you wrote down, e.g., billing, bug, or feature request. Marily Nika's Building AI-Powered Products (Ch. 1, the "Traditional AI" subsection) describes traditional AI systems as "designed to perform specific tasks through rule-based or pattern recognition systems". Routing fits that description exactly. It is one specific task, and a model learns the pattern from labeled examples. Nika also writes that GenAI "does not replace the tasks handled by traditional AI" (Ch. 1). This is the point teams skip. An LLM can do routing, because you can put the labels in a prompt and ask for one back. The question is not whether it can. The question is what you pay for the flexibility you are not using. A prompted LLM is a general tool doing a bounded job, and you pay general-tool prices for it on every request. P95 latency means the time under which 95 percent of requests finish. It is the number to watch here, because routing sits inline in the support flow and a slow tail shows up as tickets that stall. ## Work the arithmetic on the compute gap Here is a derived comparison for one 300 token ticket. These numbers come from parameter counts, not from measurement, so treat them as estimates of scale rather than benchmarks. - Option A is a fine-tuned encoder classifier with about 66 million parameters, which is about 130 MB at 16 bit weights. A forward pass costs roughly 2 operations per parameter per token. For 300 tokens that is 66 million times 2 times 300, which is about 40 billion operations. - Option B is a prompted 8B model such as [Bonsai 8B](https://huggingface.co/prism-ml/Bonsai-8B-gguf). The prompt holds instructions, the label list, and the ticket, so call it 500 tokens. Prefill costs 8 billion times 2 times 500, which is about 8 trillion operations, plus about 16 billion more for each label token it writes. Divide the two. 8 trillion over 40 billion is 200. The prompted LLM spends about 200 times the compute of the classifier on the same ticket, and this count ignores attention overhead, so the measured gap can be larger. On fixed hardware, compute per request is what you pay in latency and in money, so the 200 times shows up on both of those axes. The [cost modeling guide](https://yourwildcard.ai/docs/technical-guides/cost-modeling.md) walks through turning an operation count into a dollar figure for your own hardware. Accuracy is the axis people expect the LLM to win, and on a bounded label set it often does not. The classifier trained on your tickets has seen your label boundaries, e.g., where your team draws the line between a bug and a feature request. The prompted LLM has seen the label names and one sentence of instructions. It also has a failure mode the classifier cannot have, because it can output text that is not one of your labels, and you have to write parsing code for that case. ## Know when the LLM wins The classifier is not always the answer, and the cases where it loses are specific. - Your labels change weekly. Retraining a classifier on every label change costs more engineering time than editing a prompt. The LLM absorbs a new label list in one deploy. - You have no labeled tickets yet. A classifier needs training data. A prompted LLM routes on day one, and it can label the backlog that trains its own replacement. - The routing rule needs reasoning over the ticket, e.g., "route to legal if the customer implies a regulator is involved", where the signal is an implication rather than a phrase the classifier can learn from a few hundred examples. If none of these describe your task, the [intelligence density](https://yourwildcard.ai/docs/concepts/intelligence-density.md) page gives the general form of this argument, which is capability per unit of compute. For routing, the small model has the capability you need at a small fraction of the compute. ## Try it This takes about 30 minutes on a laptop CPU or Apple silicon. No GPU is needed. The LLM step downloads a model file of a few GB. 1. Pull 100 tickets with known correct queues from your help desk export into a CSV with two columns, text and label. 2. Build the classifier side with embeddings and nearest centroid, which needs no training run: ```bash pip install sentence-transformers numpy ``` Embed every ticket, average the embeddings per label to get one centroid per queue, and assign each ticket to its nearest centroid. Hold each ticket out of its own centroid, or split 80 to 20, so the accuracy number is honest. Wrap the loop in timing calls and record the median and P95 per ticket. 3. Build the LLM side with a local runner: ```bash brew install llama.cpp llama-cli -hf prism-ml/Bonsai-8B-gguf \ -p "Classify this ticket as billing, bug, or feature-request. Ticket: Label:" -n 4 ``` Loop it over the same 100 tickets, parse the output back to a label, and count any unparseable output as wrong. Record the same timing numbers. 4. Fill the table. Six cells, all from measurement: | | Accuracy | Median latency | Est. cost per 1,000 tickets | |---|---|---|---| | Classifier | | | | | Prompted LLM | | | | For cost, use the [cost modeling guide](https://yourwildcard.ai/docs/technical-guides/cost-modeling.md) to convert your measured seconds of compute into a dollar estimate, and label it as an estimate. 5. Write one sentence recommending an architecture, and name the column that decided it. ## Check yourself - **Your comparison table has six cells. Where did each number come from?** Expected answer: from running both systems on the same 100 tickets, not from a vendor page or from intuition. The cost cells are derived from measured time and are labeled as estimates. - **The LLM scored 3 points higher on accuracy. Do you pick it?** Expected answer: not from that number alone. Check the latency and cost columns, check whether 100 tickets can resolve a 3 point gap, and check whether the LLM's unparseable outputs were counted as errors. - **A teammate says the classifier will not keep up when the team adds queues. What do you ask?** Expected answer: how often queues actually change. If it is quarterly, a retrain per quarter is cheap. If it is weekly, that is one of the named cases where the LLM wins. - **Which subsection of Nika's Chapter 1 covers the model family that routing belongs to?** Expected answer: the "Traditional AI" subsection, because routing is one specific task learned through pattern recognition. ## Next steps - Read the [cost modeling guide](https://yourwildcard.ai/docs/technical-guides/cost-modeling.md) to turn your measured latencies into cost per ticket. - Read the [intelligence density](https://yourwildcard.ai/docs/concepts/intelligence-density.md) page for the general rule behind this comparison. - For the four groups of AI that frame this choice, read [is generative AI the same thing as AI](/blog/is-generative-ai-the-same-as-ai). When you can do this, you can run a classifier versus LLM bake-off on a routing task and select the architecture from evidence. --- --- title: "How many clicks to first inference? Auditing your docs like a funnel" description: "Treat your quickstart as a conversion funnel: count every click, copy-paste, and decision between landing and a first successful inference, then cut the count." audience: ml-product-team pillar: end-user-case-study book: transcript-theme chapter_ref: "Theme 4: Time-to-value as the docs metric" status: published last_reviewed: 2026-07-07 canonical_url: https://yourwildcard.ai/blog/clicks-to-first-inference-docs-funnel --- # How many clicks to first inference? Auditing your docs like a funnel Treat your quickstart as a conversion funnel: count every click, copy-paste, and decision between landing and a first successful inference, then cut the count like you would cart abandonment. Watching one real end user's first session, where they stall and what they open in another tab, beats any internal review of the same pages. ## Measure time-to-value, not page views A conversion funnel is a sequence of steps where some users drop out at each step. E-commerce teams count how many shoppers add an item to the cart and then leave before paying, and they redesign the steps where the most shoppers leave. The theme this post draws on applies the same method to docs. The metric for a quickstart is not how many people viewed the page. The metric is how long a new user takes to get from the docs landing page to a first successful inference, and how many of them give up before it works. Page views hide the problem. A quickstart with high traffic and a broken step looks healthy in analytics, because analytics counts arrivals and not completions. Time-to-value counts completions. If a user cannot get one working output in their first session, most of them will not come back for a second one. ## Count every step between landing and a working output Walk your own quickstart and log three kinds of steps. - A click is any navigation, e.g., following a link, opening a signup form, or switching to another page to find a value the docs assume you have. - A copy-paste is any command or code block the user must move into a terminal or editor. - A decision is any point where the docs offer options and the user must pick one, e.g., pip or conda, CPU or GPU, which model size to download. Decisions are the most expensive kind of step. A click costs seconds. A decision the user is not equipped to make costs minutes, because the user leaves your page to research it, and some never come back. Every fork in a quickstart should either have a stated default or be removed. ## Work the funnel for one quickstart Here is the arithmetic on a made-up but realistic audit. The counts below are inputs to the method, not measurements of any real project. Suppose a colleague who has never seen the project walks the quickstart while you log steps. The log shows 14 clicks, 5 copy-pastes, and 3 decisions, and the session takes 21 minutes. - Creating an account and an API key takes 5 of the 14 clicks and 8 of the 21 minutes, because the user must sign up, verify an email, and return to the docs. - One copy-paste fails with a missing dependency error, and the fix takes 6 minutes because the user searches for it in another tab. - One decision, picking a model size, takes 3 minutes because the docs list four sizes with no default. Now cut the funnel where it loses the most. Removing the key requirement for the first call cuts 5 of 14 clicks, which is about 36 percent of the clicks and 8 of the 21 minutes. Adding the missing dependency to the install command cuts 6 more minutes. Naming a default model size cuts 3 more. The same quickstart now costs 9 clicks and an estimated 4 minutes, and the estimate is marked as one because you have not timed the new version yet. The next session with a fresh user gives you the measured number. A local-first quickstart is one way to remove the signup step entirely. A model published as open weights, e.g., [Bonsai 8B on Hugging Face](https://huggingface.co/prism-ml/Bonsai-8B-gguf), can be downloaded and run without an account on your service, so the first inference needs no key at all. Watch the user's other tabs. Every time they leave your docs to search for an error, sign up for something, or look up a term, that is a stall your page caused. The tab switches are the funnel exits, and you only see them by watching a session, not by rereading the page. ## Watch one real user before you redesign anything An internal review of a quickstart finds typos. A watched session finds abandonment. The people who wrote the docs cannot stall on them, because they already know which option to pick and which error to ignore. One recorded session with a real end user shows you the three worst stalls in half an hour, and no amount of internal rereading produces that list. Ask the user to narrate while they work, and do not help them unless they are fully stuck. The places where you want to interrupt and explain are the places the docs must change. ## Try it Time a first-inference walk. Budget 30 minutes. Recruit a colleague who has never seen the project, or walk it yourself in a clean environment if no one is available. A colleague is better, because you cannot stall on your own docs. No GPU is needed if your quickstart runs a small quantized model on CPU. If your quickstart requires a GPU, log the hardware setup as part of the funnel, because your users pay that cost too. First, start a timer and a log file. ```bash date +%s > /tmp/funnel-start touch /tmp/funnel-log.md ``` Second, walk from your docs landing page to a first successful inference, e.g., the path that starts at [running your first inference](https://yourwildcard.ai/docs/getting-started/first-inference.md). Append one line to the log for every click, copy-paste, and decision, and mark every stall with the reason. ```bash echo "click: opened install page" >> /tmp/funnel-log.md echo "STALL 4min: pip error, searched fix in new tab" >> /tmp/funnel-log.md ``` Third, stop the timer and compute the totals. ```bash echo $(( $(date +%s) - $(cat /tmp/funnel-start) )) seconds grep -c "click" /tmp/funnel-log.md grep "STALL" /tmp/funnel-log.md ``` Fourth, rank the three longest stalls and write one proposed cut next to each, e.g., "add the dependency to the install command" or "state a default model size". ## Check yourself 1. What were your numbers? Expected answer: a click count and a minute count, e.g., "14 clicks and 21 minutes to first inference". If you have no numbers, you reviewed the docs instead of walking them, so run the session again with the log open. 2. Which single step cost the most time, and what fraction of the total was it? Expected answer: a named step with derived arithmetic, e.g., "account and key setup took 8 of 21 minutes, which is about 38 percent". 3. Why does a decision cost more than a click? Expected answer: a click takes seconds, but a decision without a stated default sends the user off the page to research options, and some users never return. ## Next steps - [Running your first inference](https://yourwildcard.ai/docs/getting-started/first-inference.md) is the funnel this post asks you to walk and time. - [Who is your developer ecosystem actually for?](/blog/who-is-your-developer-ecosystem-for) shows how to check whether the docs you audit serve the segment you say you want. When you can do this, you can instrument a docs funnel with click and time metrics and identify the top abandonment points. --- --- title: "How do I compare per-token API costs to renting my own GPUs?" description: "Compute total cost both ways over a week of real traffic, tokens times price versus GPU hours times rate, and add engineering time to the dedicated side." audience: ml-product-team pillar: end-user-case-study book: inference-engineering chapter_ref: "Ch. 7, sec. 7.4.2 (pp. 201-203)" status: published last_reviewed: 2026-07-07 canonical_url: https://yourwildcard.ai/blog/compare-api-costs-to-renting-gpus --- # How do I compare per-token API costs to renting my own GPUs? Do not reverse engineer a per-token price from GPU rates. Compute the total cost both ways over at least a week of real traffic. On the API side, multiply tokens by price. On the dedicated side, multiply GPU hours by the hourly rate and add engineering time. Dedicated inference turns cost from a straight line of usage into a function of deployment choices such as batch size. ## Compute the total cost, not a per-token price Kiely's Inference Engineering (Ch. 7, sec. 7.4.2) says the API bill is easy. It is "a price per million tokens times the number of tokens you use" (p. 201). Double the tokens and you double the bill. The tempting shortcut is to divide a GPU's hourly rate by the tokens it serves per hour and get your own per-token price to compare against the API's. Kiely warns against this. Given the number of variables on the dedicated side, and the gap between input and output pricing, he says it is more productive to convert your token usage into a total cost and compare totals, instead of trying to reverse engineer a per-token price from what you pay for GPUs (p. 202). The reason is that a dedicated deployment has no single per-token price. Kiely names the variables that move it: - Batch sizing. A deployment tuned for throughput packs many requests together, so each token costs less. A deployment tuned for latency does not. Our [batching guide](https://yourwildcard.ai/docs/technical-guides/batching.md) explains the tradeoff. - Traffic patterns. A GPU bills by the hour whether your traffic saturates it or not. - Sequence lengths. Long inputs and outputs change how many requests fit on each GPU. So the method is one comparison with two totals. Both totals cover the same window of real traffic, and Kiely says that window should be at least a week to smooth out daily and weekly usage cycles (p. 202). ## Work the book's example Kiely works both totals for one billing window (Fig. 7.14 and 7.15). The API side has 1,000 million input tokens at $1.25 per million and 500 million output tokens at $10 per million. That is $1,250 plus $5,000, so $6,250. The dedicated side has 1,600 GPU hours at $3.50 per hour, which is $5,600. The raw compute comparison favors dedicated by $650. But Kiely (p. 203) says the cost of engineering time spent building and maintaining the inference system should be added to the GPU cost to form the total cost of ownership. If an engineer whose time costs $15,000 a month spends even ten percent of it on the deployment, that adds $1,500 and flips the comparison back to the API. The API side carries no such line because the provider does that work. Two more notes on the arithmetic. First, output tokens are $5,000 of the $6,250 API bill, so your input to output ratio moves the API total more than the input price does. Second, the 1,600 GPU hours are paid hours, not busy hours. If your traffic only saturates the fleet half the time, you need roughly twice the paid hours for the same tokens, and the dedicated total in this example becomes $11,200. That single assumption swings the answer by more than every other line combined. ## Vary your weakest assumption Your estimate is only as good as its most fragile input, so test each one at half and at double its value: - Utilization. Paid hours divided into busy hours. Halving it doubles the dedicated compute total. - Throughput per GPU. Tokens served per hour at your sequence lengths and batch sizes. A vendor benchmark number is an estimate until you measure it on your own traffic. - Engineering time. Teams new to running GPUs usually underestimate this line. Whichever input moves the total most when doubled is your dominant assumption. Write it down next to the comparison, because that is the number to measure before you trust the answer. ## Try it Budget under 30 minutes. You do not need a GPU for this exercise, because the dedicated side is an estimate on paper. 1. Pull one week of real token usage from your API provider's usage dashboard, split into input and output tokens. 2. Compute the API cost from your provider's current prices. 3. Pick one GPU SKU and its hourly rental rate. Assume a quantized 8B model such as [Bonsai 8B](https://huggingface.co/prism-ml/Bonsai-8B-gguf), which fits on a single 24 GB card in its 4-bit form. 4. Estimate GPU hours for the week and run the comparison: ```python in_tok_m, out_tok_m = 230.0, 115.0 # YOUR week, millions of tokens in_price, out_price = 1.25, 10.0 # $ per million tokens gpu_rate = 3.50 # $ per GPU hour tps = 2500 # output tokens/sec, an ESTIMATE util = 0.5 # busy hours / paid hours eng_week = 375 # $ of engineering time per week api = in_tok_m * in_price + out_tok_m * out_price hours = (out_tok_m * 1e6 / tps) / 3600 / util dedicated = hours * gpu_rate + eng_week print(f"API ${api:,.0f} vs dedicated ${dedicated:,.0f} ({hours:,.0f} GPU hours)") for name, factor in [("util x2", 2), ("util /2", 0.5)]: h = hours / factor print(f"{name}: dedicated ${h * gpu_rate + eng_week:,.0f}") ``` 5. Note which assumption dominates. Rerun with `tps`, `util`, and `eng_week` each doubled and halved, and record which one moves the dedicated total most. ## Check yourself - Why does Kiely say to compare totals instead of per-token prices? Expected answer, because a dedicated deployment has no single per-token price. Its effective price depends on deployment variables such as batch size, so only a total over the same traffic window is a fair comparison. - Why a week of traffic instead of a day? Expected answer, because usage moves on daily and weekly cycles, and a single day gives a total that does not represent your load. - In the book's example, what does adding $1,500 of engineering time per month do? Expected answer, it erases the $650 compute advantage and the API wins the total cost comparison. - Which single assumption moves your own estimate most, and by how much at 2x? Expected answer, one named input, usually utilization, and the dollar swing from your own sensitivity run. ## Next steps - [Cost modeling](https://yourwildcard.ai/docs/technical-guides/cost-modeling.md) walks through the full comparison spreadsheet, including cache discounts and input to output ratios. - [Batching](https://yourwildcard.ai/docs/technical-guides/batching.md) explains why batch size is the main lever on the dedicated side's throughput. - [At what token volume does running my own GPU beat a per-token API?](/blog/token-volume-where-own-gpu-beats-api) turns this comparison into a single crossover number. When you can do this, you can produce a two-sided cost comparison grounded in a week of real traffic, with a sensitivity note on its dominant assumption. --- --- title: "What content do runtimes, MSPs, and inference studios need that end users never read?" description: "Ecosystem members read the integration surface: API guarantees, extension points, version matrices. End users read outcomes. Publish only one and gaps follow." audience: ml-product-team pillar: ecosystem-player book: transcript-theme chapter_ref: "Theme 2: End users vs ecosystem members (CNCF-style segmentation)" status: published last_reviewed: 2026-07-07 canonical_url: https://yourwildcard.ai/blog/content-for-runtimes-msps-inference-studios --- # What content do runtimes, MSPs, and inference studios need that end users never read? Ecosystem members build on your project, so they read the integration surface, e.g., API stability guarantees, extension points, version support matrices, and packaging conventions. End users consume outcomes, e.g., worked deployment examples, runbooks, and compliance mappings. When you publish only one kind, you tell the other audience it is not welcome, and the gap shows up later as missing adoption or missing contributions. ## Split your readers by what they do with your project The transcripts behind Theme 2 use the segmentation the CNCF uses for its projects. The CNCF is the foundation that hosts Kubernetes, and it splits every project's audience into end users and ecosystem members. The split is not about job title. It is about what the reader does with your project. - An end user runs your project to get a result for their own organization. A hospital team deploying a model on its own servers is an end user. - An ecosystem member builds a product or a service on top of your project and sells the result to someone else. A runtime that adds your model format, an MSP that operates your stack for its clients, and an inference studio that tunes deployments for hire are all ecosystem members. The two groups read different documents because they answer to different people. The end user answers for one deployment, so they want the shortest safe path to a working system. The ecosystem member answers for many customers at once, so they want to know what they can depend on across versions and what will break under them. ## Write the integration surface for ecosystem members An ecosystem member never asks how to deploy your project once. They ask what they can promise their own customers. That means they read four kinds of content that an end user skips. - **API stability guarantees.** State which interfaces are stable, which are experimental, and how much notice you give before a breaking change. An MSP writes contracts on top of these promises. - **Version support matrix.** State which versions of your project work with which versions of the runtimes and formats around it. For example, a matrix entry might say which [vLLM](https://yourwildcard.ai/docs/ecosystem/vllm.md) releases serve a given model format. - **Extension points.** Document where a third party can plug in, e.g., a custom scheduler, a new quantization format, or a billing hook, and which of those points you promise not to move. - **Packaging conventions.** State how artifacts are named, versioned, and checksummed. A public example is the [Bonsai 8B GGUF model card](https://huggingface.co/prism-ml/Bonsai-8B-gguf), which lists the exact files and quantization variants an integrator can pin to. End users do not read these documents, and that is fine. The end user reads the runbook, the worked example, and the compliance mapping. If your published content is all runbooks, the runtime engineer who wants to add your format has nothing to build against, so they either guess or leave. ## Work through one announcement split two ways Suppose you announce a new quantized release of a public 8B model. The end user version of the announcement says what got faster and links a runbook. The MSP version has to answer a different question, which is how much validation work this release creates. An MSP typically supports several environments at once. Say they cover 3 minor versions of the serving stack, 2 GPU generations, and 2 quantization variants. The combinations they must validate are 3 x 2 x 2 = 12 environments per release. If your announcement includes a support matrix that promises compatibility for the latest 2 minor versions only, the MSP can drop one minor version, and the count falls to 2 x 2 x 2 = 8 environments. That removes 4 validation runs per release. Put a cost on it. If one validation run takes about 2 engineer hours, which is an estimate and not a benchmark, the matrix saves 4 x 2 = 8 hours per release. At 12 releases a year that is 8 x 12 = 96 engineer hours. One table in your announcement does that. The end user never sees the table and never misses it. Do not publish a support matrix you do not test. An ecosystem member turns your matrix into promises to their own customers, so an untested entry in your matrix becomes a broken contract in theirs. ## Publish both kinds or accept the gap The transcripts show what happens when a project publishes for only one audience. A project with deep integrator docs and no runbooks gets contributions but few direct deployments, because end users bounce off the first page. A project with polished runbooks and no stability guarantees gets deployments but few integrations, because no runtime or MSP will build on interfaces that might move without notice. Neither gap announces itself. It shows up as a flat line in the metric you were not watching. So decide on purpose which audiences you serve, and if the answer is both, give each one a named home in your docs rather than mixing the two in one page. ## Try it This takes about 25 minutes and needs no GPU or code, only a text editor. 1. Pick one feature announcement you have shipped or plan to ship. 2. Take 10 minutes and draft a one page outline for an MSP that will integrate the feature. Include the sections an integrator needs, e.g., stability guarantees, the version matrix, packaging changes, and the deprecation timeline. 3. Take 10 minutes and draft a one page outline for an end user that will deploy the feature. Include the sections a deployer needs, e.g., a worked example, hardware requirements, and a rollback step. 4. Put the outlines side by side and mark every section that appears in both. If your docs live in a repo, you can diff the two files: ```bash diff --side-by-side msp-outline.md end-user-outline.md ``` 5. Count the sections that do not overlap at all, and write one sentence per divergent section saying which audience needs it and why. The self check is the overlap count. The two outlines should share a headline but fewer than half their sections. If they overlap heavily, you have not segmented, you have written one outline twice. ## Check yourself - **An MSP asks for your deprecation policy and you do not have one written. Which content type is missing?** Expected answer: an API stability guarantee. The MSP cannot write customer contracts on top of interfaces with no stated notice period. - **In the worked example, why did the support matrix save the MSP 4 validation runs per release?** Expected answer: the matrix promised compatibility for the latest 2 minor versions, so the MSP dropped 1 of the 3 minors they were testing, and 3 x 2 x 2 = 12 fell to 2 x 2 x 2 = 8. - **Your project has strong deployments but no third party integrations. Which half of your content is likely missing?** Expected answer: the integration surface, meaning stability guarantees, extension points, the version matrix, and packaging conventions. ## Next steps - Read the [vLLM ecosystem page](https://yourwildcard.ai/docs/ecosystem/vllm.md) as an example of an integration surface document. - Compare it against the [on-prem checklist runbook](https://yourwildcard.ai/docs/runbooks/on-prem-checklist.md), which is written for end users, and note how few sections overlap. - For the buyer side of the end user segment, read [what a hospital CTO asks before running a model on-prem](/blog/hospital-cto-on-prem-ai-questions). When you can do this, you can produce audience-differentiated content plans for one technical artifact and justify each divergent section. --- --- title: "Why do data-rich companies have an AI advantage?" description: "Models commoditize and datasets do not. Proprietary data compounds in operations and offerings, and the model that uses it can be small and local." audience: ml-product-team pillar: end-user-case-study book: building-ai-products chapter_ref: "Nika, Ch. 1, competitive-advantage paragraph" status: published last_reviewed: 2026-07-07 canonical_url: https://yourwildcard.ai/blog/data-advantage-without-an-llm --- # Why do data-rich companies have an AI advantage? Data-rich companies have an AI advantage because models commoditize and datasets do not. Proprietary data compounds in two directions, in operations such as forecasting and pricing, and in offerings such as personalization and automation. A local bakery's sales history can feed a modest demand forecast model with no cloud LLM. The defensible thing is the data loop, and the model can be small, local, and boring. ## Locate the advantage in the data, not the model Marily Nika makes this point in the opening pages of Building AI-Powered Products. Organizations with a lot of their own data "stand to gain a significant competitive advantage", and she splits that advantage into two directions. The first is operations, where she gives restocking inventory and finding the right price point as examples. The second is offerings, meaning the product itself, where the data supports personalization, recommendations, and automation. The split is the useful part, because each direction compounds on its own. - Operations. Your own history predicts your own demand, your own failures, and your own prices. A competitor's model, however large, was not trained on your Tuesdays. - Offerings. Your users' behavior inside your product teaches the product what to show next, and every session adds rows no one else has. Notice what is absent from both directions. Neither one says anything about model size. The paragraph is about who owns the rows, not who rents the biggest model. Anyone can call the same frontier API you can, so the API is not an advantage. The rows are. ## Work the bakery example Here is the worked example, with every number marked for what it is. Take a bakery with three years of point of sale history and 40 products. That is roughly 1,095 days times 40 products, or about 43,800 rows of date, product, and units sold. No other business has these rows. Now take one product and use illustrative numbers to see what the rows can do. Suppose the croissant costs $1.10 to make and sells for $3.50, so each unsold one loses $1.10 and each missed sale loses $2.40 of margin. Suppose the baker bakes a flat 60 every day, and the history shows a weekday mean of 42 sold and a Saturday mean of 65. The flat bake count has a derivable cost. - Weekdays: 60 baked minus 42 sold is 18 wasted per day. Five weekdays times 18 times $1.10 is $99 per week. - Saturday: 65 wanted minus 60 baked is 5 missed sales, times $2.40 margin, is $12 per week. That is about $111 per week, or roughly $5,700 per year, on one product. These are illustrative inputs, not benchmarks, but the arithmetic is exact once you plug in your own numbers. The point of the example is what closes the gap. A per product, per weekday average closes most of it. Gradient boosted trees on the full table, with weather and local events joined in, close more. Both run on the shop's own laptop CPU in seconds. No LLM appears anywhere in the loop, and the loop still compounds, because every day of sales makes next week's forecast better. The offerings direction works from the same table. The bakery can text regulars when the thing they usually buy is about to come out of the oven. That is a lookup over purchase history, not a generative model. If the owner later wants plain language on top, e.g., a drafted weekly supplier order email, a small local model such as [Bonsai 8B](https://huggingface.co/prism-ml/Bonsai-8B-gguf) can do that on the same machine. The language model is a convenience at the edge of the loop. It is not the moat. ## Test whether the advantage is real The test is one question. Could a competitor replicate this with a bigger model but without our data? If yes, you did not find a data advantage. You found a feature anyone can rent. Apply it to the bakery. A competitor with a frontier API but no POS history cannot forecast this shop's Saturday croissant demand, because the signal is in rows only this shop has. The advantage passes. Now apply it to "we summarize industry news for our customers with an LLM". A competitor with a bigger model and no data from you produces the same summaries. That fails the test, and no amount of prompt work fixes it. The test also sizes the model for you. Once the advantage is confirmed to live in the data, pick the smallest model that can extract it, because the model is the replaceable part. Per group averages, linear models, and gradient boosted trees cover most operations uses. The [intelligence density](https://yourwildcard.ai/docs/concepts/intelligence-density.md) page covers how to think about capability per unit of compute, and the [cost modeling guide](https://yourwildcard.ai/docs/technical-guides/cost-modeling.md) covers what each model class costs to run. A bigger model is not a safer default here. It costs more per prediction, and it does not add rows to your table. When the advantage is the data, extra model capacity past what the data supports is pure spend. ## Try it This is the exercise from the top of the post, and it takes about 25 minutes with no GPU. 1. Take 10 minutes and list datasets your organization uniquely owns. One line each: what it records, roughly how many rows, and since when. "Uniquely owns" excludes anything a competitor can buy or scrape. 2. Pick the one dataset with the most rows and the longest history. 3. Write one operations use for it, meaning a decision your own team makes better with it, e.g., restocking or pricing. 4. Write one product use for it, meaning something a user sees, e.g., a recommendation or an automated step. 5. For each use, name the smallest model class that could exploit it: a group average, a linear model, gradient boosted trees, a small classifier, or a small local language model. 6. Run both uses through the test. Could a competitor replicate this with a bigger model but without our data? Rewrite any use that fails until it passes or admit it is not a data advantage. If your dataset is tabular sales or usage data, you can get a first baseline in the same sitting: ```bash python3 -m venv venv && source venv/bin/activate pip install pandas python3 - <<'EOF' import pandas as pd df = pd.read_csv("sales.csv", parse_dates=["date"]) # columns: date, product, units df["dow"] = df["date"].dt.day_name() print(df.groupby(["product", "dow"])["units"].mean().round(1)) EOF ``` That printout is a per product, per weekday demand forecast. It is the zero model baseline that any fancier model has to beat. ## Check yourself - **Did both of your uses pass the replication test?** Expected answer: yes, and you can say which rows the competitor lacks. If a use passed only because your prompt is clever, it failed. - **Is your smallest model actually small?** Expected answer: for most operations uses, an average or a tree model, not an LLM. If you wrote "LLM" for a forecasting or pricing use, redo step 5 and justify what the language model adds. - **What does one more month of data do to each use?** Expected answer: the forecast or recommendation gets measurably better, which is what compounding means here. If more data changes nothing, the advantage is not in the data. - **What would the bakery's yearly waste be if the weekday mean were 50 instead of 42?** Expected answer: 60 minus 50 is 10 wasted per weekday, times 5 days, times $1.10, is $55 per week, or about $2,860 per year, before the Saturday term. ## Next steps - Read the [intelligence density](https://yourwildcard.ai/docs/concepts/intelligence-density.md) concept page for how to match model capability to the task instead of defaulting to the largest model. - Use the [cost modeling guide](https://yourwildcard.ai/docs/technical-guides/cost-modeling.md) to price the smallest model that passed your test against the API you almost reached for. - For a case where the generative model was the wrong tool and the data loop was the right one, read [why GenAI is the wrong tool for personalization](/blog/genai-wrong-tool-for-personalization). When you can do this, you can identify a genuine proprietary data advantage and pair it with a plan for the smallest model that can use it. --- --- title: "Do inference optimizations like quantization and speculation stack together?" description: "Not automatically. Quantizing the KV cache helps disaggregation, but big batches starve speculation. Finding a balanced set by config search is the work." audience: inference-engineer pillar: foundational-concept book: inference-engineering chapter_ref: "Ch. 5, intro (pp. 119-120)" status: published last_reviewed: 2026-07-07 canonical_url: https://yourwildcard.ai/blog/do-inference-optimizations-stack --- # Do inference optimizations like quantization and speculation stack together? Not automatically. The techniques interact, and the interactions cut both ways. Quantizing the KV cache helps disaggregation, while big batches starve speculative decoding of the idle compute it needs. The job is to compose a balanced set through patient experimentation. One practitioner tried 77 configurations before finding a combination that doubled throughput. Config search is the actual work. ## Learn the five families and how they touch Kiely's Inference Engineering (Ch. 5 intro) groups the acceleration techniques into five families: quantization, speculation, caching, parallelism, and disaggregation. Each one trades a different resource. Quantization cuts memory and memory traffic. Speculation spends spare compute to produce more tokens per step. Caching skips repeated prefill work. Parallelism and disaggregation split work across GPUs. Because they trade different resources, they meet in the same budget. Kiely writes that "sometimes techniques are symbiotic and sometimes they are incompatible." His two examples from the chapter intro are the ones in the lead. Quantizing the KV cache shrinks the state that a disaggregated setup has to move between prefill and decode machines, so those two techniques help each other. Increasing the batch size uses up the compute that speculative decoding needs, so those two fight. The chapter's goal statement is worth keeping in mind whenever you tune a server. The engineer's job, Kiely writes, is "a balanced set of optimizations that delivers more than the sum of its parts." A balanced set is not the union of every technique turned to maximum. ## Work through why batching starves speculation Here is the compute budget behind the batch and speculation conflict, with round illustrative numbers. Take an 8B model at FP16, so about 16 GB of weights. Each decoded token costs about 2 FLOPs per parameter, which is 16 GFLOPs per token. Suppose the GPU reads memory at 1 TB/s and computes at 100 TFLOPS. These are estimates chosen for easy arithmetic, not benchmarks. ```text memory time per decode step = 16 GB / 1 TB/s = 16 ms compute time at batch 1 = 16 GFLOPs / 100 TFLOPS = 0.16 ms compute time at batch 100 = 100 x 16 GFLOPs / 100 TFLOPS = 16 ms ``` At batch 1, the step takes 16 ms because of memory, and the compute units are busy for 0.16 ms of it. That idle 15.84 ms is what speculative decoding spends. Verifying several draft tokens in one step multiplies the compute per step but barely changes the memory time, so the extra tokens are close to free. At batch 100, compute time has grown to match memory time and there is no slack left. Now every draft token verified is compute the batch needed for regular decoding, so speculation slows the server down instead of speeding it up. Both settings help alone. Together, one cancels the other. This is the shape of interaction the chapter warns about, and quantization adds a third dial, because 4-bit weights cut the memory time to about 4 ms and move the crossover point again. ## Search configurations instead of trusting intuition Kiely tells a story from a Baseten hackathon. An engineer tuning an autocomplete model for code tried 77 configurations with a handwritten script before finding a combination that doubled tokens per second for a customer, and the winning combination was not obvious. The lesson is not the number 77. The lesson is that he wrote a script and swept, rather than reasoning his way to one config and shipping it. The chapter also says the size of your search should match your traffic. Techniques like KV-aware routing and dynamic disaggregation only pay off with many GPUs serving one model, and tuning is not a one-time task, because traffic changes. For a small deployment, a sweep over quantization level, cache precision, batch limits, and draft length covers most of what you can turn. ## Try it This takes about 25 minutes. You need llama.cpp, two quantizations of one model, e.g., the Q4_K_M and Q8_0 files of [Bonsai 8B](https://huggingface.co/prism-ml/Bonsai-8B-gguf), and a small GGUF draft model that shares the same tokenizer for the speculation runs. No GPU is required, but note what your machine is. On a CPU or a laptop there is little idle compute during decode, so speculation often loses there, and seeing that is part of the exercise. Sweep 8 configurations: two weight quantizations, two KV cache types, and speculation off or on. ```bash PROMPT="Write a 300 word explanation of how a hash table works." for MODEL in Bonsai-8B-Q4_K_M.gguf Bonsai-8B-Q8_0.gguf; do for KV in f16 q8_0; do for DRAFT in 0 8; do if [ "$DRAFT" = "0" ]; then SPEC=""; else SPEC="-md draft.gguf --draft-max 8"; fi echo "=== $MODEL kv=$KV draft=$DRAFT ===" llama-cli -m "$MODEL" -p "$PROMPT" -n 256 --temp 0 --seed 1 \ --cache-type-k "$KV" -fa on $SPEC 2>&1 | grep "eval time" done done done ``` Record the decode tokens per second from each run in a small table. Then look for a pair of settings where each helps alone but the combination is worse than either. A common find is that speculation helps the Q8_0 model more than the Q4_K_M model, because the 4-bit weights already shrank the memory time that speculation was hiding compute inside. If your draft model rarely agrees with the target, lower `--draft-max` and rerun, since rejected drafts are pure wasted compute. ## Check yourself 1. Name one pair of techniques from the chapter that help each other and one pair that conflict. Expected: quantizing the KV cache helps disaggregation because there is less cache state to hand between prefill and decode. Large batches conflict with speculation because they consume the idle compute that verification needs. 2. Why does speculative decoding get cheaper as decode gets more memory bound? Expected: in a memory bound step the compute units are idle most of the step, so verifying draft tokens uses time the GPU was wasting anyway. 3. Your server runs 4-bit weights at batch 64, and adding a draft model made throughput drop. What happened? Expected: at that batch size and precision, decode is near compute bound, so draft verification competes with regular decoding instead of filling idle compute. 4. A teammate benchmarks each optimization alone, keeps every winner, and ships all of them together without a combined test. What is the flaw? Expected: the gains are not independent. Two settings that each help can conflict in combination, so the combined config has to be measured itself. ## Next steps - [Speculative decoding guide](https://yourwildcard.ai/docs/technical-guides/speculative-decoding.md) for draft models, acceptance rates, and verification cost. - [Quantization guide](https://yourwildcard.ai/docs/technical-guides/quantization.md) for number formats and which components to quantize first. - [How does speculative decoding work?](/blog/how-speculative-decoding-works) for the mechanism this post treats as one dial among several. When you can do this, you can design a small configuration sweep and detect a negative interaction between two optimizations. --- --- title: "When should I use a draft model vs EAGLE vs n-gram speculation?" description: "Draft-target is easiest but has the highest overhead, EAGLE is the general-purpose go-to, and n-gram lookup wins on code tasks where output mirrors input." audience: inference-engineer pillar: ecosystem-player book: inference-engineering chapter_ref: "Ch. 5, secs. 5.2.1-5.2.4 (pp. 131-136)" status: published last_reviewed: 2026-07-07 canonical_url: https://yourwildcard.ai/blog/draft-model-vs-eagle-vs-ngram-speculation --- # When should I use a draft model vs EAGLE vs n-gram speculation? Draft-target speculation is the easiest to set up but has the highest overhead. EAGLE, a purpose-built speculator that reads the target model's hidden states, is the general-purpose choice. N-gram lookup beats both on code-like tasks where the output mirrors the input. No speculator is best everywhere, so match the algorithm to the workload. ## Start with draft-target when you need speculation today Draft-target is the original method. A small draft model guesses the next few tokens, and the target model validates them in one forward pass. Kiely's Inference Engineering (Ch. 5, sec. 5.2.1, pp. 131 to 132) says it is a good choice when you want something quick to set up without any training or fine-tuning. You pick a model from the same family as the target, at least ten times smaller by parameter count so the tokenizers and behaviors match, and you are running. The cost is overhead. Kiely calls draft-target the method with the most overhead of any speculative decoding approach. The engine must hold the draft model's weights, activations, and KV cache in memory. It must spend compute on draft model prefill. It must also coordinate two models so they do not compete for resources, though engines like TensorRT-LLM handle that orchestration for you. ## Reach for EAGLE as the general-purpose default An off-the-shelf draft model such as Qwen 0.5B was built to be a good standalone model on cheap hardware, not to speculate for a large target on a server GPU. Kiely notes that these draft models run inefficiently and get a low acceptance rate (sec. 5.2.3, p. 133). EAGLE fixes this by training a speculator from scratch for the job. During inference the target model accumulates context in its hidden states between layers, and a normal draft model never sees any of it. EAGLE takes three of those hidden states as input, one from an early layer, one from a middle layer, and one from a late layer, and outputs draft tokens. It is often under one billion parameters, drafts sequences of up to eight tokens, and gets a very high acceptance rate. It also attaches to the same module as the target model, so one forward pass runs both and there are no round trips to the CPU like draft-target needs. Kiely calls EAGLE the go-to speculation algorithm for general use among engineers who have the means to train EAGLE heads, and inference engines support it well. Its predecessor Medusa (sec. 5.2.2) grafted two to four extra decoder heads onto the target model itself, but its draft length and acceptance rate were limited, and Kiely says it is not widely used in production today. ## Use n-gram lookup when the output copies the input N-gram speculation has no draft model at all. During prefill, the engine builds a dictionary of token sequences it saw in the input. During decode, it looks up the last generated token in the dictionary, and any stored suffix becomes the draft. The target validates that draft as normal. Drafting is a dictionary lookup, so it is close to free, and the sequences can exceed ten tokens where EAGLE tops out around eight. The catch is that the dictionary only contains sequences from the input. Acceptance is high only when the output repeats the input. Kiely writes that n-gram speculation is mostly used for code completion and code revision, and that "within this specific domain, it easily outperforms EAGLE" (sec. 5.2.4, p. 135). A close relative, Lookahead Decoding, builds the dictionary during inference instead, which handles less repetitive context but spends extra compute to generate the n-grams. ## Work the copy-ratio arithmetic Here is a worked example. Every number in it is illustrative, not a benchmark. Suppose a code revision task produces 400 output tokens, and 300 of them sit inside spans copied straight from the input file. With n-gram lookup and a draft length of 10, assume near-total acceptance inside copied spans and near-zero acceptance elsewhere. Each pass inside a copied span yields 10 accepted drafts plus 1 generated token, so the 300 copied tokens take about 300 / 11, or 28 passes. The 100 novel tokens get no useful drafts, so they take about 100 passes. That is roughly 128 forward passes instead of 400, about a 3.1x cut, and the drafting itself cost almost nothing. Now run EAGLE on the same task with a draft length of 4 and a uniform 0.7 acceptance per draft token. A draft at position k only survives if every earlier draft did, so the expected accepted count is 0.7 + 0.49 + 0.343 + 0.24, about 1.77 tokens, and each pass yields about 2.77 tokens. The 400 tokens take about 144 passes, plus the cost of running the speculator itself. N-gram wins here. Flip the workload to open prose with a copy ratio near zero. The n-gram dictionary almost never fires, so n-gram speculation gives back plain decode at 400 passes. EAGLE still gives about 144. Same two algorithms, opposite winner, and the copy ratio is the whole difference. ## Try it This takes about 20 minutes on a laptop, CPU is fine, no GPU needed. You need llama.cpp built from source, because prompt lookup decoding ships as the `llama-lookup` example binary. Any GGUF model works, e.g., [Bonsai 8B](https://huggingface.co/prism-ml/Bonsai-8B-gguf) or a Qwen2.5 7B quant. Put a function of 40 or more lines in a file, then wrap it in an editing prompt: ```bash cat > /tmp/revise.txt <<'EOF' Revise this Python code to add type hints. Return the full revised code. EOF # Baseline, no speculation llama-cli -m bonsai-8b-q4_k_m.gguf -f /tmp/revise.txt -n 512 --temp 0 \ 2>&1 | grep -Ei "tokens per second|eval time" # Prompt lookup (n-gram) decoding, same prompt llama-lookup -m bonsai-8b-q4_k_m.gguf -f /tmp/revise.txt -n 512 --temp 0 \ 2>&1 | grep -Ei "drafted|accept|tokens per second|eval time" ``` Record tokens per second for both runs, and the accepted and drafted counts that `llama-lookup` logs. Then rerun both commands with a prompt that has no code to copy, e.g., "Write a short story about a lighthouse keeper," and record the numbers again. The revision prompt should show a clear speedup and the story prompt should show little or none, because the story output does not repeat the input. ## Check yourself 1. Why does draft-target have the highest overhead when the draft model is the smallest part of the setup? Expected answer: the engine must hold a second model's weights, activations, and KV cache in memory, spend compute on draft prefill, and orchestrate two models per step. EAGLE avoids most of this by running in the target's own forward pass, and n-gram has no model at all. 2. What does EAGLE receive as input that a normal draft model never sees, and why does that raise acceptance? Expected answer: hidden states from an early, a middle, and a late layer of the target. Those states describe what the target is about to generate, so the speculator predicts the target's actual next tokens instead of guessing from text alone. 3. Your workload is summarizing news articles into three fresh sentences. Which speculator do you expect to win, and why not n-gram? Expected answer: EAGLE or another learned speculator. The summary rephrases the input rather than copying spans from it, so the copy ratio is low and the n-gram dictionary rarely produces an accepted draft. 4. In the worked example, why did n-gram beat EAGLE on the code revision task despite EAGLE's better model? Expected answer: 75 percent of the output was copied spans where n-gram acceptance was near 1 with a draft length of 10, and its draft cost was near zero. EAGLE's 0.7 acceptance over length 4 yields fewer tokens per pass and it still pays to run a speculator. ## Next steps - [Speculative decoding guide](https://yourwildcard.ai/docs/technical-guides/speculative-decoding.md) for the shared draft-and-validate mechanism and engine support for each algorithm. - [llama.cpp ecosystem page](https://yourwildcard.ai/docs/ecosystem/llama-cpp.md) for building from source and where the example binaries live. - [What determines whether speculative decoding actually speeds things up?](/blog/what-determines-speculative-decoding-speedup) for the three factors that this post's arithmetic is built on. When you can do this, you can match speculation algorithms to workload characteristics and demonstrate the n-gram win on an editing task. --- --- title: "What's the fastest way to run Whisper locally — streaming or batch?" description: "Streaming whisper.cpp with small chunks minimizes latency to the first word. Batch runs over whole files maximize audio hours per GPU hour. Same weights." audience: inference-engineer pillar: ecosystem-player book: inference-engineering chapter_ref: "Ch. 6, secs. 6.3.1-6.3.2 (pp. 163-165); Ch. 1, sec. 1.2.2" status: published last_reviewed: 2026-07-07 canonical_url: https://yourwildcard.ai/blog/fastest-way-to-run-whisper-locally --- # What's the fastest way to run Whisper locally — streaming or batch? It depends on which speed you are buying. Streaming whisper.cpp with small chunks cut at pauses in speech gives the lowest latency to the first word, so it fits live dictation. Batch runs over whole files finish the most audio hours per hour of compute, so they fit backlogs. The weights are the same, and only the deployment changes. ## Name the metric before you tune anything "Fastest" hides two different numbers. For live use, the number is round-trip time from when you stop speaking to when the words appear. Kiely's Inference Engineering (sec. 6.3.1) suggests aiming for 200 milliseconds per chunk, because that is the average human reaction time. For a backlog of files, the number is how many hours of audio you transcribe per hour of wall clock. The book (sec. 6.3.2) measures this as the Real-Time Factor, or RTF. An RTF of 2X means you transcribe one hour of audio in 30 minutes, and Kiely reports that a deployment tuned for long files can reach an RTF of 1000X on server GPUs. This is the online versus offline split from Chapter 1 (sec. 1.2.2) applied to speech. One configuration cannot win both numbers, so pick the metric first. Our post on [online versus offline inference](/blog/online-vs-offline-inference) covers the general version of this trade. ## Configure streaming when a person is waiting Kiely (sec. 6.3.1) makes a point that surprises people. Streaming is not a feature of the model runtime. The runtime still transcribes fixed chunks of audio. Streaming is built one layer up, where a server holds a connection open, a voice activity detection model watches the incoming audio, and the server cuts a chunk whenever the speaker pauses. A voice activity detection model, or VAD, is a small model that flags which spans of audio contain speech. Each chunk goes to Whisper as a normal request and the text comes back over the connection. Chunk size is the main lever. The model cannot transcribe a word before the audio for it exists, so latency to the first word is at least the chunk length plus the inference time on that chunk. A 2 second chunk can show words about 2 seconds after you start talking. A 10 second chunk cannot beat 10 seconds no matter how fast the GPU is. Small chunks cost accuracy. Whisper reads audio in windows of up to 30 seconds, and a word cut in half at a chunk boundary is often transcribed wrong. Kiely notes that when chunks run in order on the same machine, you can pass the previous chunk's text in as a prefix for the next chunk, which restores some of the lost context. ## Configure batch when the files are already on disk For a backlog, no one is watching a single request, so let every chunk wait its turn. Kiely (sec. 6.3.2) describes the pipeline for long files. A VAD model first removes silence and cuts the audio at pauses rather than at fixed intervals, because a fixed interval risks cutting words in half. The chunks then run in parallel, packed into large batches so the hardware stays busy, and the transcripts are stitched back together by timestamp. Parallel chunks give up the previous-chunk prefix trick. The book's answer is to check each chunk's output for repeated words and abnormal words per minute, which are the usual signs of a hallucinated transcript, and to re-run only the suspect chunks. Kiely writes that in practice these checks "obviate the need for passing a previous sequence as a prefix" (Inference Engineering, sec. 6.3.2). ## Work the overhead arithmetic on one example The times below are made up to show the arithmetic. Measure your own before deciding anything. Suppose each chunk pays 0.3 seconds of fixed overhead for model startup, padding, and scheduling, on top of the transcription itself. - With 2 second chunks, one hour of audio is 1,800 chunks, and the overhead alone is 1,800 x 0.3 = 540 seconds, which is 9 minutes per audio hour. - With 10 second chunks, one hour of audio is 360 chunks, and the overhead is 360 x 0.3 = 108 seconds, which is under 2 minutes per audio hour. - With 30 second chunks, the overhead is 120 x 0.3 = 36 seconds per audio hour. The transcription work is the same in every row, so the small chunks pay 15 times the overhead of the large ones. That is the whole trade in one table. Small chunks buy latency to the first word and pay for it in throughput and accuracy. Large chunks buy throughput and pay for it in waiting. The [batching guide](https://yourwildcard.ai/docs/technical-guides/batching.md) explains the same mechanism for text models. ## Try it You can measure the trade yourself in about 25 minutes with [whisper.cpp](https://yourwildcard.ai/docs/ecosystem/whisper-cpp.md) and a microphone. No GPU is needed. The base model runs at usable speed on a laptop CPU, and whisper.cpp uses the GPU on Apple silicon automatically. Build with the microphone example enabled and fetch a model. The streaming tool needs the SDL2 library, e.g., `brew install sdl2` on a Mac. ```bash git clone https://github.com/ggml-org/whisper.cpp && cd whisper.cpp cmake -B build -DWHISPER_SDL2=ON && cmake --build build -j ./models/download-ggml-model.sh base.en ``` Read one fixed paragraph aloud three times, once per chunk size. The `--step` flag sets how many milliseconds of new audio each inference step consumes. ```bash ./build/bin/whisper-stream -m models/ggml-base.en.bin --step 2000 --length 6000 ./build/bin/whisper-stream -m models/ggml-base.en.bin --step 5000 --length 10000 ./build/bin/whisper-stream -m models/ggml-base.en.bin --step 10000 --length 10000 ``` For each run, record two things. First, the round-trip latency, which you can time from the end of a sentence to the text appearing. Second, the word errors, which you count by comparing the output to your paragraph. Then try the VAD mode, which cuts at your pauses instead of at a fixed interval: ```bash ./build/bin/whisper-stream -m models/ggml-base.en.bin --step 0 --length 30000 -vth 0.6 ``` Expect the 2 second runs to show text fastest and to garble more words, mostly at chunk boundaries. Expect the VAD run to fix many of those errors, because pauses are places where no word is being cut. ## Check yourself 1. What is the floor on latency to the first word for a 5 second chunk, even on infinitely fast hardware? Expected answer: 5 seconds plus the inference time, because the audio for a word has to exist before the model can transcribe it. 2. In your microphone test, at which chunk size did accuracy degrade, and what is the mechanism? Expected answer: your own smallest size, and the mechanism is that a chunk boundary cuts words in half and drops the surrounding context that Whisper would otherwise use. 3. Why does a batch pipeline cut chunks with a VAD model instead of every N seconds? Expected answer: per Kiely (sec. 6.3.2), fixed intervals risk cutting words in half, while cutting at silence keeps each word intact. 4. Parallel batch transcription cannot pass the previous chunk's text as a prefix. What replaces it? Expected answer: automatic checks on each chunk for repeated words and abnormal words per minute, then re-running the suspect chunks with a higher temperature or smaller chunks. ## Next steps - [whisper.cpp](https://yourwildcard.ai/docs/ecosystem/whisper-cpp.md) covers installing and configuring the tool used above. - [Batching](https://yourwildcard.ai/docs/technical-guides/batching.md) explains why large batches raise throughput and latency together. - [What's the difference between online and offline inference?](/blog/online-vs-offline-inference) works the same trade with cost arithmetic for a transcription backlog. When you can do this, you can configure whisper.cpp for a latency target or a throughput target and state the accuracy trade at each chunk size. --- --- title: "When should I use a few-step image model?" description: "Few-step models generate in eight or fewer steps, 80 to 90 percent faster with noticeably lower quality. Decide the trade with users, not benchmarks." audience: ml-product-team pillar: end-user-case-study book: inference-engineering chapter_ref: "Ch. 2, sec. 2.3.2 (p. 59)" status: published last_reviewed: 2026-07-07 canonical_url: https://yourwildcard.ai/blog/few-step-image-models-for-realtime-features --- # When should I use a few-step image model? Few-step image models generate in eight or fewer denoising steps. They run 80 to 90 percent faster than base models, and their quality is noticeably lower. That trade fits a use that needs low latency and can accept lower quality, e.g., a real-time filter. This looks like an infrastructure decision, but it is a product decision about where your quality bar sits, so decide it with users, not benchmarks. ## Know what a few-step model is A normal image generation model refines random noise into an image over 30 to 50 denoising steps. Kiely's Inference Engineering (Ch. 2, sec. 2.3.2, p. 59) notes that these steps are the most time consuming part of image generation. A few-step model is trained to produce a high resolution image in eight or fewer steps. The book states these models are "80 to 90 percent faster out of the box" (Inference Engineering, p. 59), and that their output quality is noticeably lower. The book lists two ways teams build them. - **Latent consistency.** Train a model to predict the target latent directly, then repeat the prediction two to four times to improve quality. - **Distillation.** Train a small model to imitate a larger one in fewer steps, using adversarial or progressive distillation. Distillation is the more common method today. When a new base model like FLUX or Qwen Image ships, the open image model community publishes distillations of it, so you rarely have to train one yourself. ## See where the speedup comes from The speedup is not a clever kernel or a faster GPU. The model simply runs fewer forward passes, and you can derive the saving from step counts alone. In a standard model, each denoising step runs two forward passes, one conditioned on the prompt and one without, combined by the guidance scale. So a 50 step generation runs about 100 forward passes. A few-step model that uses 8 steps runs at most 16 passes even if it keeps the two-pass scheme, and many few-step models drop the second pass entirely. Going from 100 passes to 16 removes 84 percent of the passes. Since nearly all generation time is spent in these passes, that lines up with the book's 80 to 90 percent figure. To turn that into latency, measure your own base model once. As an example, if a base generation takes 10 seconds on your GPU, an estimate for the distilled version is 1 to 2 seconds. That is the difference between a feature where the user waits and a feature that responds while they watch. The same arithmetic drives cost, because a request that runs 16 passes instead of 100 needs about a sixth of the GPU time. The [cost modeling guide](https://yourwildcard.ai/docs/technical-guides/cost-modeling.md) shows how to carry that into a per-image price. ## Decide with your quality bar, not a benchmark The book's recommendation is direct. If you have a latency sensitive use case where quality is less important, such as real-time generative filters, consider few-step models. The hard part is knowing whether your use case is quality tolerant, and no public benchmark can answer that, because the bar depends on what your users are doing. Two examples show the split. - A camera filter in a social app has a bar of "fun and recognizable". Users see the image for seconds and speed is the feature. A few-step model clears this bar. - A product image generator for an online store has a bar of "good enough to represent the merchant's product". Artifacts in hands, text, or fine detail fail the bar, so the base model's extra steps are the feature. So the question "should I use a few-step model" is a product question wearing infrastructure clothes. Answer it the way you would answer any quality question, by putting outputs in front of people and asking whether they clear the bar, not by comparing model names or step counts. The exercise below is that test in miniature. ## Try it Run a small trade study in under 30 minutes. You need a GPU with 8 GB or more of VRAM or an Apple Silicon Mac. On CPU only, the base model runs take several minutes each, so budget more time. ```bash pip install diffusers torch transformers accelerate ``` ```python import time, torch from diffusers import AutoPipelineForText2Image device = "cuda" if torch.cuda.is_available() else "mps" prompts = [ "a golden retriever wearing sunglasses", "a bowl of ramen on a rainy windowsill", "a neon sign that says OPEN", "an astronaut planting a garden", "a watercolor map of a small island", ] base = AutoPipelineForText2Image.from_pretrained( "stabilityai/stable-diffusion-2-1", torch_dtype=torch.float16).to(device) turbo = AutoPipelineForText2Image.from_pretrained( "stabilityai/sd-turbo", torch_dtype=torch.float16).to(device) for i, p in enumerate(prompts): t0 = time.time() base(p, num_inference_steps=50).images[0].save(f"base_{i}.png") t1 = time.time() turbo(p, num_inference_steps=1, guidance_scale=0.0).images[0].save(f"turbo_{i}.png") t2 = time.time() print(p, "base:", round(t1 - t0, 1), "s, turbo:", round(t2 - t1, 1), "s") ``` sd-turbo is a distillation of Stable Diffusion 2.1, so the two models share a lineage and the comparison is fair. Record the average speedup across the five prompts. Then rename the ten files to random names, state your use case's quality bar in one sentence, and have a colleague vote on each image without knowing which model made it. Ask one question only, whether the image clears the bar. Your recommendation should cite both numbers, e.g., "12x faster, and 4 of 5 turbo images cleared the bar for a chat sticker feature". Also name the use case where the vote would flip, e.g., "for product listings, 0 of 5 would clear it". ## Check yourself 1. Where does a few-step model's speedup come from? Expected answer: it runs far fewer forward passes. A 50 step generation runs about 100 passes because each step runs two, while an 8 step model runs 16 or fewer, and generation time is almost entirely these passes. 2. What does the trade cost you, according to the book? Expected answer: output quality is noticeably lower than a traditional image model. 3. Your team is picking a model for an id photo tool and a teammate proposes a few-step model because "it benchmarks 10x faster". What is missing from that argument? Expected answer: the quality bar. An id photo tool likely cannot tolerate the quality drop, and only a blind evaluation against the use case's bar can show whether the faster model clears it. 4. Name the two methods the book gives for creating few-step models and say which is more common. Expected answer: latent consistency and distillation, and distillation is more common today. ## Next steps - Read the [quantization guide](https://yourwildcard.ai/docs/technical-guides/quantization.md) for a second speedup that stacks with fewer steps, since both cut latency in different ways. - Read the [cost modeling guide](https://yourwildcard.ai/docs/technical-guides/cost-modeling.md) to turn the pass count arithmetic into a per-image cost. - Read [why is image generation compute-bound when the models are small](/blog/why-image-generation-is-compute-bound) for the forward pass math this post builds on. When you can do this, you can run a latency and quality trade study with blind evaluation and match a model class to what your use case tolerates. --- --- title: 'How to fine tune Ternary Bonsai 27B' description: 'Choose between MLX QLoRA adapters and the unpacked FP16 checkpoint when adapting PrismML Ternary Bonsai 27B.' audience: local-ai-builder pillar: bonsai-27b status: published last_reviewed: 2026-07-15 canonical_url: https://yourwildcard.ai/blog/fine-tune-ternary-bonsai-27b --- # How to fine tune Ternary Bonsai 27B Start with a LoRA or QLoRA adapter instead of changing all 27 billion ternary weights. MLX LM supports adapter training on quantized models, and the packed Bonsai checkpoint already loads with stock mlx-lm 0.31.3. Treat this as an adapter experiment until a short training run, held-out evaluation, and adapter inference test pass on the exact checkpoint. ## The practical local path When the model is quantized, MLX LM uses QLoRA: the packed base stays frozen and training updates small adapter matrices. Begin with short examples, a small batch, and a short sequence length on a 24 GB Mac. Keep the adapter separate for the first evaluation. Fusing and requantizing can change the ternary operating point and should not be treated as equivalent to PrismML's quantization-aware training recipe. ```bash uv tool install 'mlx-lm[train]' mlx_lm.lora \ --model prism-ml/Ternary-Bonsai-27B-mlx-2bit \ --train \ --data ./data \ --adapter-path ./adapters/bonsai-27b \ --iters 100 ``` ## When the unpacked model helps The unpacked repository contains FP16 safetensors for stock Hugging Face training frameworks. It is useful for cloud LoRA, SFT, DPO, or GRPO experiments that need a conventional checkpoint, but PrismML says it is about 54 GB and provides none of the packed model's local memory benefit. PrismML has not published a complete 27B ternary training and repacking pipeline in its public demo repository. A full-weight fine tune is therefore a research project, not a routine export step. Ask PrismML for its QAT and packing workflow before promising a new 1.71 bits-per-weight release. ## Questions people ask ### Can I fine tune the packed 2 bit weights directly? QLoRA keeps the packed base frozen and trains adapters around it. That is different from updating and re-ternarizing every base weight. ### Will a fused adapter remain a true ternary model? Do not assume so. Fusion and requantization need a measured quality check and a compatible packing path. ## Sources - [MLX LM LoRA and QLoRA documentation](https://github.com/ml-explore/mlx-lm/blob/main/mlx_lm/LORA.md) - [Ternary Bonsai 27B MLX model card](https://huggingface.co/prism-ml/Ternary-Bonsai-27B-mlx-2bit) - [Ternary Bonsai 27B unpacked FP16 model card](https://huggingface.co/prism-ml/Ternary-Bonsai-27B-unpacked) - [PrismML Bonsai demo repository](https://github.com/PrismML-Eng/Bonsai-demo) - [Qwen3.6 27B base model card](https://huggingface.co/Qwen/Qwen3.6-27B) ## Related Bonsai 27B lessons - [How good is Ternary Bonsai 27B?](/blog/ternary-bonsai-27b-quality-benchmarks) - [Qwen3.6 27B versus Ternary Bonsai 27B](/blog/qwen36-27b-vs-ternary-bonsai-27b) - [Ternary Bonsai 27B limitations and verification checklist](/blog/ternary-bonsai-27b-limitations) The benchmark numbers on this page describe one checkpoint, runtime, machine, and test shape. Reproduce the test on the hardware and workload you plan to use before making a product decision. --- --- title: "What's the difference between fine-tuning and distillation?" description: "Fine-tuning adapts a model to a domain with new training data. Distillation trains a small student on a large teacher's probability distributions." audience: researcher pillar: foundational-concept book: inference-engineering chapter_ref: "Ch. 1, sec. 1.3.3 (pp. 33-35)" status: published last_reviewed: 2026-07-07 canonical_url: https://yourwildcard.ai/blog/fine-tuning-vs-distillation --- # What's the difference between fine-tuning and distillation? Fine-tuning adapts a model to a domain with new training data. Distillation trains a small student model on a large teacher model's probability distributions, so the student keeps the teacher's general behavior at a lower cost. Distillation is rarer in practice. When a distill lands on a popular architecture, as DeepSeek-R1 did with Llama and Qwen, it inherits that architecture's performance tooling. ## Separate the two by their training signal The clean way to tell these apart is to ask what the model learns from, not what it is for. In fine-tuning, you take a pretrained model and continue training it on new data from your domain. Philip Kiely's *Inference Engineering* (Ch. 1, sec. 1.3.2) defines it as taking a foundation model and adapting it to a specific use case by introducing new data. The training signal is a set of input and output pairs that you supply. The model's architecture and size stay the same, and only the weight values change. In distillation, you take a large model, called the teacher, and use it to train a smaller model, called the student. The signal is different. Kiely writes in section 1.3.3 that distillation "shows the student model the teacher model's actual probability distributions", not just its final answers. A probability distribution here is the full set of scores the teacher assigns to every possible next token, not only the token it picked. The student learns to match those scores, so it absorbs how the teacher weighs its options, not just what the teacher said. This also separates distillation from fine-tuning on synthetic data. If you generate text with a large model and fine-tune a small model on that text, the small model only sees final answers. That is fine-tuning. Distillation gives the student the richer signal underneath each answer. The goals differ in the same way. Fine-tuning narrows a model to do better in one domain. Distillation copies the teacher's general behavior into a smaller model, and Kiely notes it copies the flaws along with the strengths. ## See why distillation is rarer Kiely is blunt about usage. "Distillation sees substantially less real-world use than fine-tuning" (sec. 1.3.3). One reason he gives is how model families are made. When a lab releases a family of sizes, the small models are usually trained independently rather than distilled from the large one, because the lab does not want the large model's biases to cap the small ones. Distillation earns its place when a lab only trains one very large model and wants to make it accessible. That is the DeepSeek-R1 story. In January 2025, DeepSeek released R1 at 671B parameters, and alongside it they released distilled versions built on the most popular open architectures of the time, Llama 3 and Qwen 2.5. Kiely reports that the distills showed similar reasoning behavior to the main model, with worse benchmark scores. The choice of target architectures did as much for the distills as the training did. Kiely points out in section 1.3 that inference engines vary in how well they support each architecture, so a model on a popular architecture gets the existing performance work for free. A distill onto Llama runs everywhere Llama runs, with every kernel and quantization recipe that already exists for Llama. A distill onto a novel architecture would have started with none of that. ## Work the arithmetic on the R1 distills Here is what the size gap means in memory, using 2 bytes per parameter for FP16 weights. See the [quantization guide](https://yourwildcard.ai/docs/technical-guides/quantization.md) for where these byte counts come from. - DeepSeek-R1 at 671B parameters needs about 671 x 2 = 1,342 GB for weights alone. No single GPU holds that. At 80 GB per GPU, weights alone need at least 17 GPUs, so serving it means a multi-GPU node before any KV cache. - The Llama 8B distill needs about 8 x 2 = 16 GB at FP16, which fits one 24 GB card. A 4-bit quantization brings it near 4.5 GB, which fits a laptop. - That is a parameter reduction of 671 / 8, which is about 84x. The distill does not keep all of the teacher's intelligence, and the book says so. What it keeps is the teacher's behavior, e.g., R1's habit of writing out a long reasoning trace before answering, at a memory cost 84x smaller. ## Pick the right tool for your goal Match the method to what you want the small model to be. - You want a small model that is strong at one narrow task, and you have labeled data for it. Fine-tune. Kiely's example is text to SQL, where a fine-tuned model of a few billion parameters can match general coding models hundreds of times its size on that one task, because the domain is cleanly scoped. - You want a small model that behaves like a specific large model across general use, and you have access to that teacher. Distill. - You want the result to run fast on common tools. Whichever method you pick, put it on a popular architecture, because the inference support follows the architecture. ## Try it Compare a distill against a plain base model of the same size and watch the teacher's fingerprint appear. You need llama.cpp, which can pull models straight from Hugging Face. The two Q4 files total about 9 GB of downloads. Each model runs on a Mac with 16 GB of memory or a GPU with 8 GB of VRAM, and no large GPU is needed. The whole loop takes under 30 minutes on a normal connection. Run the distill first. ```bash llama-cli -hf unsloth/DeepSeek-R1-Distill-Qwen-7B-GGUF:Q4_K_M \ -p "A train leaves at 3:40 pm and the trip takes 2 hours 35 minutes. When does it arrive?" ``` Then run a base model of the same scale on the same prompt. Qwen2.5-7B-Instruct is the closest match, since the distill was built on it. [Bonsai 8B](https://huggingface.co/prism-ml/Bonsai-8B-gguf) also works as a same-scale comparison point. ```bash llama-cli -hf Qwen/Qwen2.5-7B-Instruct-GGUF:Q4_K_M \ -p "A train leaves at 3:40 pm and the trip takes 2 hours 35 minutes. When does it arrive?" ``` Repeat with two more problems, e.g., a short logic puzzle and a request to answer in exactly three sentences. For each pair of outputs, note the shape of the response before you judge its correctness. The distill should open with a long reasoning trace inside think tags before it commits to an answer, because that is R1's behavior. The base model answers directly. You are done when you can point at a concrete output habit in the distill, such as the reasoning trace or its closing summary format, that came from the teacher and not from the 7B base. ## Check yourself 1. Both methods update a model's weights with more training. What is the one difference in the training signal? Expected answer: fine-tuning trains on input and output pairs, while distillation trains the student on the teacher's probability distributions over tokens, not just its final answers. 2. You generate 100K answers with a frontier model and train a 7B model on them. Is that distillation? Expected answer: no, that is fine-tuning on synthetic data, because the small model only sees final answers and never the teacher's distributions. 3. You need a small model for one scoped task, e.g., text to SQL, and you have labeled examples. Which method? Expected answer: fine-tuning, because the goal is domain quality, not copying a specific model's general behavior. 4. Why did DeepSeek distill R1 onto Llama and Qwen instead of a shrunken R1 architecture? Expected answer: inference engines already support those popular architectures well, so the distills could use existing performance tooling from day one. ## Next steps - [Quantization](https://yourwildcard.ai/docs/technical-guides/quantization.md) covers the other main way to shrink a model's memory cost, and it composes with both methods here. - [llama.cpp](https://yourwildcard.ai/docs/ecosystem/llama-cpp.md) explains the runtime used in the exercise and how it loads models from Hugging Face. - [What is the biggest inference optimization?](/blog/model-selection-biggest-inference-optimization) argues that picking a smaller model, by whatever method, beats any runtime trick. When you can do this, you can distinguish fine-tuning from distillation by their training signals and predict which one suits a given adaptation goal. --- --- title: "What do FlashAttention and PagedAttention actually optimize?" description: "FlashAttention removes redundant memory reads and writes, and PagedAttention pages the KV cache. Both are lossless implementation work, not new math." audience: researcher pillar: ecosystem-player book: inference-engineering chapter_ref: "Ch. 2, sec. 2.5 (pp. 67-68)" status: published last_reviewed: 2026-07-07 canonical_url: https://yourwildcard.ai/blog/flashattention-vs-pagedattention --- # What do FlashAttention and PagedAttention actually optimize? FlashAttention and PagedAttention are lossless implementation work, not new math. FlashAttention removes redundant memory reads and writes inside the attention kernel. PagedAttention splits the KV cache into pages so it no longer needs one contiguous block of GPU memory. Contrast that with new algorithms like sliding window attention, which trade quality for sub-quadratic complexity. Once you know which category a technique is in, you know whether to expect quality regressions. ## Separate implementations from algorithms Kiely's Inference Engineering (Ch. 2, sec. 2.5) argues that there are two strategies for optimizing attention. The first is to write better kernels that compute the same math with less wasted memory traffic. The second is to change the math itself so it scales in better than quadratic time. Kiely writes that implementation improvements "are lossless (do not affect quality) and make inference feasible for long sequences" on today's hardware. This split is the useful part for a researcher. A lossless implementation can never change the model's outputs beyond floating point noise, so you can adopt it without an evaluation run. A new algorithm changes what the model computes, so you have to measure quality before and after. When a vendor announces an attention speedup, your first question should be which of the two bins it falls in. ## Count the reads and writes FlashAttention removes The standard attention implementation writes its intermediate matrices to GPU memory and then reads them back one step later. Kiely works this out for a decode step with a 128 dimension head over 4,096 tokens. The score matrix S and the probability matrix P are each 4096 x 4096, which is about 32 MiB each in FP16. The book sums every read and write in the three step algorithm and gets a total memory movement of 8N² + 8Nd bytes against 4N²d + 3N² floating point operations. That works out to an arithmetic intensity of 62 operations per byte, far below the H100's ratio of about 295, so the kernel is memory bound. Here is the arithmetic on where those bytes go. With N = 4096 and d = 128, the 8N² term is about 134 MB and the 8Nd term is about 4 MB. So roughly 97 percent of the memory traffic in the naive kernel is the round trip of S and P, matrices that no one needs after the kernel finishes. FlashAttention keeps those intermediates in on-chip memory and never writes them to the GPU's main memory at all. The math is identical, and the wasted 134 MB of traffic per head is gone. The 62 ops per byte figure comes from the book's worked exercise. The 97 percent split is my own division of the two terms in that formula, not a measured benchmark. ## See what PagedAttention pages PagedAttention solves a different problem. The KV cache holds the key and value vectors for every previous token, and it grows as the model generates. Kiely notes that KV caches quickly fill GPU memory and take time to read. The classic failure is fragmentation. Each request needs a contiguous region sized for a length you do not know in advance, so the server over-reserves, and free memory ends up scattered in unusable gaps between requests. PagedAttention, the core idea inside [vLLM](https://yourwildcard.ai/docs/ecosystem/vllm.md), splits each request's KV cache into fixed size blocks and finds them through a lookup table. The cache for one request can then live in scattered blocks across the GPU. This is bookkeeping, not math. The attention output is unchanged, and the win shows up as more concurrent requests fitting on the same card. If the mechanics of the cache itself are new to you, start with the [KV cache guide](https://yourwildcard.ai/docs/technical-guides/kv-cache.md). ## Sort techniques into the two bins Kiely lists the algorithmic side of the ledger in the same section. Each of these changes what attention computes, so each can affect output quality. - Sliding window attention only attends to the previous w tokens, which turns the cost from O(N²) into O(Nw). - Linear attention replaces the quadratic softmax step with a linear time approximation. - Compressed attention periodically compresses older context and attends over the compressed form. - Multi-latent attention approximates attention in a smaller latent space. The book adds one honest caveat. When a lossy algorithm like sliding window attention is applied during training rather than bolted on at inference, the model learns around it and quality holds up well. The category still tells you what to check. A lossless implementation needs no quality evaluation, and a lossy algorithm always needs one. ## Try it llama.cpp ships FlashAttention behind a flag, so you can measure the lossless claim on your own machine in a few minutes. Use any GGUF model you already have and a long prompt file, e.g., a few thousand tokens of text. ```bash # Baseline, flash attention off ./llama-cli -m model.gguf -f longprompt.txt -n 64 -fa off # Flash attention on ./llama-cli -m model.gguf -f longprompt.txt -n 64 -fa on ``` Read the timing block that llama.cpp prints at the end of each run. The prompt eval time is your time to first token. Record the delta between the two runs, and note that the generated text should be essentially identical, because this is an implementation change and not an algorithm change. On some CPU-only or older GPU setups the flag does little or nothing, and that is itself a useful result, since kernel-level optimizations only pay off when the hardware they target is present. ## Check yourself 1. FlashAttention makes prefill faster. Can it change the model's outputs? Expected answer: no, beyond floating point noise, because it computes the same math with less memory traffic. 2. Sort these five into lossless implementation versus lossy algorithm: FlashAttention, PagedAttention, sliding window attention, linear attention, multi-latent attention. Expected answer: the first two are lossless implementations, and the last three are lossy algorithms because each changes what attention computes. 3. A serving framework claims it fits 3x more concurrent requests in the same VRAM with no quality change. Which technique in this post does that sound like, and why is the no quality change claim plausible? Expected answer: PagedAttention, because paging the KV cache changes memory layout and not the attention output. 4. In the book's decode example, memory movement is 8N² + 8Nd bytes. Which term does FlashAttention attack? Expected answer: the 8N² term, the round trip of the S and P intermediate matrices. ## Next steps - [The KV cache guide](https://yourwildcard.ai/docs/technical-guides/kv-cache.md) explains the structure that PagedAttention pages. - [The vLLM ecosystem page](https://yourwildcard.ai/docs/ecosystem/vllm.md) covers the server built around PagedAttention. - [Why is prefill compute bound and decode memory bound?](/blog/prefill-compute-bound-decode-memory-bound) covers the bottleneck analysis that explains why removing memory traffic helps at all. When you can do this, you can classify attention optimizations by category and predict whether each one can affect output quality. --- --- title: "When is GenAI the wrong tool for personalization?" description: "Recommendation is mostly traditional AI. Embedding lookups and rankers cost orders of magnitude less than generation, so a 'for you' feed rarely needs an LLM." audience: inference-engineer pillar: end-user-case-study book: building-ai-products chapter_ref: "Nika, Ch. 1, 'Personalized media' subsection" status: published last_reviewed: 2026-07-07 canonical_url: https://yourwildcard.ai/blog/genai-wrong-tool-for-personalization --- # When is GenAI the wrong tool for personalization? GenAI is the wrong tool when the job is ranking, because recommendation is largely traditional AI. Embedding lookups and rankers cost orders of magnitude less than generation, and a Netflix style "for you" feed does not need an LLM in the request path. GenAI adds new content around the feed. It does not replace the recommender, and for a feed the architecture choice dominates cost, because a 1 GB embedding service autoscales on CPUs where an LLM ranker keeps a GPU busy per request. ## Read what the book actually claims Marily Nika lists "Personalized media" under GenAI in Building AI-Powered Products (Ch. 1). She writes that platforms such as Netflix and Spotify use AI to suggest shows and music that match user preferences. It is easy to read that as "the feed is GenAI now", but the same chapter blocks that reading. Nika writes that GenAI "does not replace the tasks handled by traditional AI" and instead opens new dimensions (Ch. 1). The suggestion engine itself is the traditional part. It learns patterns from watch and listen history and picks items from a fixed catalog. Nothing is generated. The GenAI dimensions sit around that engine, e.g., a generated description of a playlist, generated artwork for a row of titles, or Spotify's AI DJ speaking between songs. So a music app has both kinds of AI in it, and an inference engineer should serve them very differently. The rest of this post works out why. ## Work the arithmetic on ranking one feed Take one feed request: rank 1,000 candidate items for one user. Compare a standard embedding recommender against an LLM asked to do the ranking. All numbers below are derived from parameter counts, not measured, so treat them as estimates of scale rather than benchmarks. Option A is the embedding path. - The 1,000 item vectors were computed offline in a nightly batch, so they cost nothing at request time. - Embedding the user profile with a 100M parameter model over a 200 token profile takes about 2 operations per parameter per token, so about 40 billion operations. You can cache this vector, because a profile changes on the scale of hours, not requests. - Scoring is 1,000 dot products over 768 dimensions, about 1.5 million operations. The whole request costs about 40 billion operations on a cold profile, and almost nothing on a cached one. Option B is an LLM ranker such as [Bonsai 8B](https://huggingface.co/prism-ml/Bonsai-8B-gguf). You put the 200 token profile and 1,000 item titles at about 15 tokens each into the prompt, about 15,200 tokens. Prefill on an 8B model costs about 2 operations per parameter per token, so about 243 trillion operations, plus decode for the ranked list. Divide the two. 243 trillion over 40 billion is about 6,000, so the LLM ranker spends roughly three to four orders of magnitude more compute per request, and the gap grows with profile caching. The LLM also returns text, so you must parse the ranking back out and handle the case where it names an item that is not in the candidate list. The embedding path returns scores by construction. ## Size the serving footprint The compute gap turns into an architecture gap. Work out the memory for the embedding service. A 250,000 item catalog at 768 dimensions and 4 bytes per value is about 768 MB. The whole scoring service, vectors included, fits in about 1 GB of RAM on a CPU node. When traffic doubles you add another cheap replica, and [batching](https://yourwildcard.ai/docs/technical-guides/batching.md) the dot products is straightforward because they have no sequential dependency. The LLM ranker needs the 8B model resident, about 16 GB at 16 bit or about 5 GB quantized, plus KV cache for a 15,200 token prompt per in-flight request. That means a GPU per replica and a much higher floor cost, and scaling means adding GPUs. The [cost modeling](https://yourwildcard.ai/docs/technical-guides/cost-modeling.md) guide walks through turning these footprints into a per request price. Run that model per 1,000 users and the embedding path should come out at least an order of magnitude cheaper on any current pricing, because the compute ratio above is three to four orders of magnitude before serving overheads. Use the LLM where generation is the job, e.g., writing the one line reason a title was picked for you. That call can run async, cached per title, off the feed's latency path. ## Try it This takes about 20 minutes on a laptop CPU. No GPU is needed, and the model download is under 100 MB. 1. Install a local embedding model: ```bash pip install sentence-transformers ``` 2. Embed 20 items and one user profile, then rank by cosine similarity: ```python from sentence_transformers import SentenceTransformer, util model = SentenceTransformer("all-MiniLM-L6-v2") items = [ "gritty crime drama set in 1970s New York", "lighthearted baking competition series", "documentary about deep sea creatures", # ... add 17 more one-line descriptions ] profile = "watches slow-burn crime dramas and true crime documentaries" item_vecs = model.encode(items) user_vec = model.encode(profile) scores = util.cos_sim(user_vec, item_vecs)[0] for score, item in sorted(zip(scores, items), reverse=True)[:5]: print(f"{score:.3f} {item}") ``` 3. Sanity check the top five against your intuition. The crime drama should beat the baking show for this profile. 4. Time the script end to end, then estimate what the same ranking costs through an LLM API at 15 tokens per item plus your profile, using the prices on your provider's page. ## Check yourself - **Roughly how much more compute does an 8B LLM ranker spend per feed request than the embedding path above?** Expected answer: about three to four orders of magnitude. The derived figures were about 243 trillion operations against about 40 billion, and the embedding side drops further with profile caching. - **Your product team says "Netflix uses GenAI for personalization, so our feed should too." What is the correct split?** Expected answer: the feed ranking is traditional AI over embeddings, and GenAI adds content around it, e.g., generated descriptions or artwork, ideally off the latency path. - **Why does the embedding recommender autoscale more cheaply than the LLM ranker?** Expected answer: the whole scoring service fits in about 1 GB of RAM and runs on CPU replicas, while the LLM needs a GPU per replica with gigabytes of weights and KV cache resident. - **Can you estimate cost per 1,000 users for both paths?** Expected answer: yes, by multiplying tokens per request by your provider's prices or by using the compute ratio, and the gap should be at least an order of magnitude in favor of embeddings. ## Next steps - Read the [batching](https://yourwildcard.ai/docs/technical-guides/batching.md) guide, because offline item embedding is a bulk workload with different tradeoffs from the live scoring path. - Read the [cost modeling](https://yourwildcard.ai/docs/technical-guides/cost-modeling.md) guide to turn the operation counts above into a per request price for your own traffic. - For how the two embedding workloads in this design differ in serving, read [serving embeddings in bulk versus live](/blog/serving-embeddings-bulk-vs-live). When you can do this, you can prototype an embedding-based recommender and quantify its cost advantage over generative ranking. --- --- title: "What does a hospital CTO actually ask before running a model on-prem?" description: "Regulated buyers evaluate in order: risk first (data residency, audit trails, liability), feasibility second, features last. Lead with capability and demos stall." audience: ml-product-team pillar: end-user-case-study book: transcript-theme chapter_ref: "Theme 3: How regulated end users evaluate local AI" status: published last_reviewed: 2026-07-07 canonical_url: https://yourwildcard.ai/blog/hospital-cto-on-prem-ai-questions --- # What does a hospital CTO actually ask before running a model on-prem? The order of evaluation is the surprise. A hospital CTO asks about risk first, e.g., where patient data lives and who is liable when the model fails. Feasibility comes second, meaning whether the model runs on hardware the hospital can own with the staff it already has. Features come last. A pitch that opens with capability answers the third question before the buyer has asked the first, and that is why demos alone stall. ## Learn the order a regulated buyer uses On-prem means the model runs on servers the hospital owns, inside its own network. The transcripts behind Theme 3 show the same pattern across regulated end users. The CTO works through three groups of questions in a fixed order, and does not move to the next group until the current one is settled. **Risk questions come first.** These are the questions the CTO must answer to a board, a regulator, or a lawyer. - Where does patient data live, and does any of it leave our network? - Who can audit what the model saw and what it produced? - If the model output harms a patient, who is liable, us or the vendor? - What happens to our data if the vendor shuts down or is acquired? **Feasibility questions come second.** These are the questions the CTO must answer to the operations team. - Does the model run on hardware we can buy and rack ourselves? - Can our current staff operate it, or does it need a hire we do not have budget for? - What breaks when the model needs an update? **Feature questions come last.** Only after the first two groups are settled does the CTO ask what the model can do, how accurate it is, and how fast it responds. The order is not a preference. It reflects who can say no. A regulator or a lawyer can kill the project at the risk stage, and no feature can save it there. So the CTO spends attention in the order the vetoes arrive. ## Work through the arithmetic the CTO runs at the feasibility stage The feasibility stage has real numbers in it, so here is a worked example. Suppose the proposal is a 70 billion parameter model quantized to 4 bits, which means each weight is stored in half a byte. - Weight memory: 70 billion parameters x 0.5 bytes = 35 GB. - Working memory for the cache and overhead: add roughly 20 percent, so plan for about 42 GB of GPU memory. - That fits on one 48 GB card, or two 24 GB cards with the model split across them. A single server with one 48 GB workstation card costs somewhere in the range of $15,000 to $30,000. That range is an estimate from public list prices in mid 2026, not a benchmark, so check current prices before you quote it. The CTO will also add a second server for failover and a fraction of an engineer to own updates and monitoring. If that fraction is a quarter of a $180,000 salary, the staffing line is $45,000 per year, which is larger than the hardware line. The staffing question is usually the one that decides feasibility, not the GPU. If you want the deeper version of this sizing arithmetic, see [how much VRAM you need to run an LLM](/blog/how-much-vram-to-run-an-llm). Do not quote accuracy or throughput numbers to a regulated buyer unless you can show how they were measured. The transcripts show CTOs treating an unsourced benchmark claim as a risk signal about the vendor, which moves the conversation backward to the risk stage. ## Reorder your pitch to match the buyer Most technical pitches and READMEs are written in the reverse order. They open with features, mention deployment in the middle, and put security and compliance in an appendix. That structure forces the CTO to read to the end before the first question gets an answer. To fix it, restructure the material so it answers the questions in the order the CTO asks them. 1. Open with the risk answers. State where data lives, what gets logged, and what the vendor is liable for. Link the audit documentation. 2. Follow with the feasibility answers. State the exact hardware requirement, the update process, and the skills needed to operate it. 3. Close with the features. By this point the reader has permission to care about them. The demo still has a place. It belongs after the risk conversation, as evidence for the feature claims, not as the opening move. ## Try it This takes about 25 minutes. 1. Take 10 minutes and write the first ten questions a hospital CTO would ask about your on-prem proposal. Write them as the CTO would say them, not as your documentation is organized. 2. Label each question R for risk, F for feasibility, or X for features. Count each label. 3. Now audit your own README or pitch deck. From the repo root, list the section headings in reading order: ```bash grep -n "^#" README.md ``` 4. Label each heading R, F, or X the same way. 5. Compare the two orderings. Note the first R heading in your README and how many X headings appear before it. ## Check yourself - **Which two sections of your README would you move to the top?** Expected answer: the two sections that carry your R labels, typically security or data handling and deployment requirements. If you cannot name two, your README does not answer the risk questions anywhere, which is a bigger finding. - **In your ten CTO questions, how many were risk questions?** Expected answer: four or more. If you wrote mostly feature questions, rewrite the list from the point of view of someone who reports to a board. - **Where does the demo belong in the restructured pitch?** Expected answer: after the risk answers, as evidence for the feature claims. ## Next steps - Read the [regulated deployment case study](https://yourwildcard.ai/docs/case-studies/regulated-deployment.md) for a full account of one buyer's evaluation. - Work through the [on-prem checklist runbook](https://yourwildcard.ai/docs/runbooks/on-prem-checklist.md) before your next proposal. - If the feasibility numbers above were new to you, start with [how much VRAM you need to run an LLM](/blog/how-much-vram-to-run-an-llm). When you can do this, you can reconstruct a regulated buyer's evaluation sequence and restructure a technical pitch to match it. --- --- title: "How many GPUs do I need to serve a large LLM?" description: "Minimum GPU count is arithmetic: precision bytes times parameters times a 1.8x KV cache multiplier, rounded up to the next instance size." audience: ml-product-team pillar: foundational-concept book: inference-engineering chapter_ref: "Ch. 5, sec. 5.4 (p. 142)" status: published last_reviewed: 2026-07-07 canonical_url: https://yourwildcard.ai/blog/how-many-gpus-to-serve-an-llm --- # How many GPUs do I need to serve a large LLM? Minimum GPU count is arithmetic you can do before touching hardware. Multiply the bytes per parameter by the parameter count, then by a KV cache multiplier of roughly 1.8x on top of weights, and round up to the next instance size. Weights fitting is not enough. The KV headroom is the part everyone forgets. ## Start from the formula in the book Philip Kiely's *Inference Engineering* (Ch. 5, sec. 5.4) gives the estimate as one line of code. The VRAM required is the precision in bits divided by 8, times the parameter count in billions, times a KV cache allocation factor, and he uses 1.8 for that factor. The result is in gigabytes. You then round up to the next available instance size, because GPUs come in fixed configurations such as one, two, four, or eight cards per node. The 1.8 multiplier is an estimate, not a law. It covers the KV cache, which is the memory the server keeps for every token in every active request, plus buffers. Long context or high batch sizes push the true need above 1.8, and short context pushes it below. It is the right starting number for a first sizing pass, and you refine it later with a load test. Kiely also explains why you cannot skip the multiplier. He writes that "It's not enough to just barely squeeze the model weights into VRAM." His example is DeepSeek-V3.1, a 671 billion parameter model. At FP8 precision, the weights alone are about 671 GB. Four B200 GPUs give 720 GB, so the weights technically load. But the KV cache often takes 80 percent or more of the VRAM that remains after weights, and 49 GB of leftover room is not enough to serve any reasonable context length or batch size. The deployment that works is a full node of eight B200s. ## Work one example end to end Here is the book's own case, with the arithmetic shown. - Precision: FP8, so 8 bits / 8 = 1 byte per parameter. - Parameters: 671 billion, so weights are about 671 GB. - KV multiplier: 671 x 1.8 = about 1,208 GB required. - Available instance sizes for the B200, which has 180 GB each: 180, 360, 720, or 1,440 GB. - 1,208 GB does not fit in 720 GB, so round up to 1,440 GB. That is eight B200 GPUs. Notice what the rounding step does. The formula said 1,208 GB, but you cannot buy 1,208 GB. You buy the next size up, and the extra 232 GB is not waste. It becomes room for more KV cache, which means more concurrent users or longer context per user. Kiely notes that even midsize models are often served above the minimum GPU count for exactly this reason. More cache room gives better latency per user. One caution for the product plan. Once a model needs more than one GPU, the GPUs have to split the model between them and talk to each other on every step, which the book covers in the same section under model parallelism. The formula tells you the minimum count. It does not promise that four cards run four times faster than one. ## Try it This exercise needs no GPU. It is a calculator and two spec sheets, and it takes under 30 minutes. Compute VRAM for these three combinations with the formula (bits / 8) x params in billions x 1.8. ```text 1. Bonsai 8B at FP16: (16/8) x 8 x 1.8 = 28.8 GB 2. A 70B model at FP8: (8/8) x 70 x 1.8 = 126 GB 3. DeepSeek-V3.1 at FP8: (8/8) x 671 x 1.8 = 1,207.8 GB ``` Now check each against a real spec sheet and pick an instance. 1. Look up the NVIDIA L40S (48 GB) and H100 (80 GB) spec pages, and the [Bonsai 8B model card](https://huggingface.co/prism-ml/Bonsai-8B-gguf). Does 28.8 GB fit on one L40S? Yes, with about 19 GB of margin. 2. Does 126 GB fit on one H100? No. Two H100s give 160 GB, so the answer is a two GPU instance with about 34 GB of margin. 3. Does 1,208 GB fit on four B200s at 720 GB? No. Round up to eight B200s at 1,440 GB. For each one, write down the fit or no fit call and the margin in GB. You have done the exercise correctly when your margin includes the KV cache, not just the weights. ## Check yourself 1. A 32B model at FP16. What does the formula say, and does it fit on one 80 GB H100? Expected answer: 2 x 32 x 1.8 = 115.2 GB, so no. It needs two H100s (160 GB), which leaves about 45 GB of margin. 2. Your team quantizes that same 32B model to FP8. What changes? Expected answer: 1 x 32 x 1.8 = 57.6 GB, so it now fits on one H100 with about 22 GB of margin. Halving the precision halved the whole requirement, because the multiplier applies to weights. 3. A teammate says "the weights are 671 GB and four B200s have 720 GB, so we are covered." What is wrong? Expected answer: the 49 GB left over has to hold the KV cache for every active request, and that is not enough to serve real traffic. The formula with the 1.8 multiplier says 1,208 GB, which needs eight B200s. 4. Where does the 1.8 come from, and when is it wrong? Expected answer: it is Kiely's starting estimate for KV cache and buffers on top of weights. It is too low for very long context or heavy concurrency and too high for short prompts at batch size one, so you confirm it with a load test before you commit to hardware. ## Next steps - [Hardware requirements](https://yourwildcard.ai/docs/getting-started/hardware-requirements.md) maps common GPUs and instance sizes to the model sizes they can hold. - [KV cache](https://yourwildcard.ai/docs/technical-guides/kv-cache.md) explains what the 0.8x of headroom in the multiplier is actually storing. - [How much VRAM do I need to run an LLM?](/blog/how-much-vram-to-run-an-llm) does the same budget at single GPU scale, with the per token cache math written out. When you can do this, you can size a GPU deployment for a given model and precision using the capacity formula. --- --- title: "How much VRAM do I need to run an LLM?" description: "VRAM must hold the model weights plus roughly 50 percent headroom for KV cache and activations. Forgetting that headroom causes OOM crashes at long context." audience: inference-engineer pillar: foundational-concept book: inference-engineering chapter_ref: "Ch. 3, sec. 3.1.2 Memory and Caches" status: published last_reviewed: 2026-07-07 canonical_url: https://yourwildcard.ai/blog/how-much-vram-to-run-an-llm --- # How much VRAM do I need to run an LLM? VRAM must hold the model weights plus roughly 50 percent headroom for the KV cache and activations, so weights fitting is not enough. Forgetting KV cache headroom is the most common capacity planning failure. It shows up as OOM crashes, or as mysteriously slow inference once your users send long context. ## Start from the rule in the book Philip Kiely's *Inference Engineering* (Ch. 3, sec. 3.1.2) states the rule directly. The VRAM should hold "the model weights, plus at least 50 percent headroom for KV cache", and he adds that long context and high batch sizes need more than that. He also names the two failure modes. If the weights alone do not fit, the model fails to load with an out of memory error. If the weights fit but the headroom does not, inference is slow or crashes later with the same error. The second failure is the one that surprises people. The model loads fine, the demo works, and then the server crashes in production. The gap between "it loads" and "it runs" is the KV cache. ## Compute the weights first The weights number is easy. Multiply the parameter count by the bytes per parameter. - FP16 uses 2 bytes per parameter. A 7B model needs about 14 GB for weights. - FP8 uses 1 byte per parameter. The same model needs about 7 GB. - A 4-bit quantization such as Q4_K_M uses roughly 0.56 bytes per parameter, so the same model needs about 4 GB. Many people stop here, see that 14 GB fits on a 24 GB card, and conclude that they are done. That is the mistake. ## Add the KV cache The KV cache stores the attention keys and values for every token in the context, so the server does not recompute them on each new token. Its size grows linearly with context length. The formula per token is 2 (one for keys, one for values) times the layer count, times the KV head count, times the head dimension, times the bytes per value. Here is the worked example for a Llama-2 style 7B model with 32 layers, 32 KV heads, a head dimension of 128, and FP16 cache values. - Per token: 2 x 32 x 32 x 128 x 2 bytes = 524,288 bytes, which is 0.5 MB per token. - At 2K context: 2,048 x 0.5 MB = 1 GB. - At 32K context: 32,768 x 0.5 MB = 16 GB. Now put the budget together for a 24 GB card. - FP16 weights at 32K context: 14 GB + 16 GB = 30 GB. This does not fit. It OOMs even at batch size 1. - Q4 weights at 32K context: 4 GB + 16 GB = 20 GB. This fits, but only 4 GB is left for activations and everything else, and a second concurrent request doubles the cache demand. Two things soften this in practice, and you should know both. First, models with grouped query attention have fewer KV heads. A model with 8 KV heads instead of 32 cuts the cache to 0.125 MB per token, so 32K context needs 4 GB instead of 16 GB. Second, most servers can quantize the cache itself to 8 bits, which halves it again. But the shape of the problem does not change. The cache still grows linearly with context and with batch size, and at some context length it no longer fits on any card. The arithmetic above is for one request. Ten concurrent requests at 32K context each need ten separate KV caches. This is why a server that handles one long chat fine can crash the moment real traffic arrives. Kiely also notes in the same section that memory bandwidth, not memory size, is the bottleneck for decode at low to medium batch sizes. Size decides whether you can run the model at all, and bandwidth decides how fast the tokens come out. This post is about the first question. ## Try it Watch the failure happen on your own machine. You need llama.cpp and any 7B or 8B GGUF file. The [Bonsai llama.cpp recipe](https://yourwildcard.ai/docs/build-and-run/bonsai-llamacpp.md) gets you a working setup if you do not have one. Load the model at a small context and note the memory use. On macOS, watch the process in Activity Monitor or with `top`. On Linux with a GPU, use `nvidia-smi`. ```bash llama-server -m ./model-q4_k_m.gguf -c 2048 ``` Record the resident memory. Then stop the server and reload at a long context. ```bash llama-server -m ./model-q4_k_m.gguf -c 32768 ``` The difference between the two readings is the KV cache allocation plus a small compute buffer. Now keep doubling the context until the load fails or the machine starts swapping. That context length is the failure point for this model on this hardware. Before each run, predict the memory use with the formula above, and check your prediction against the reading. The whole loop takes under 30 minutes. ## Check yourself 1. A 7B model at FP16 on a 24 GB card. How much room is left for KV cache after the weights load? Expected answer: 24 - 14 = 10 GB, minus a compute buffer, so 32K context at 0.5 MB per token (16 GB) does not fit. 2. The same model quantized to Q4. Does 32K fit now? Expected answer: weights drop to about 4 GB, so about 20 GB is free and the 16 GB cache fits, with little margin. 3. Your model uses grouped query attention with 8 KV heads instead of 32. What happens to the cache budget? Expected answer: the cache shrinks by 4x, to 0.125 MB per token, so 32K context needs about 4 GB. 4. A teammate says "the model is 14 GB and the card is 16 GB, so we are fine." What do you ask? Expected answer: at what context length and what batch size, because the KV cache is not in that 14 GB. ## Next steps - [KV cache](https://yourwildcard.ai/docs/technical-guides/kv-cache.md) explains where the cache comes from and why servers keep it at all. - [Hardware requirements](https://yourwildcard.ai/docs/getting-started/hardware-requirements.md) maps common cards and Macs to the model sizes they can hold. - [What are the three layers of an inference stack?](/blog/three-layers-of-an-inference-stack) places this memory budget inside the runtime layer, which is one of three layers you have to get right. When you can do this, you can produce a weights plus KV memory budget for any model and hardware pair and identify the failure point before you ever load the model. --- --- title: "How does speculative decoding generate more than one token per forward pass?" description: "Decode is memory-bandwidth-bound, so a small speculator drafts tokens and the target validates them in one pass. It lifts TPS and ITL, never TTFT." audience: inference-engineer pillar: foundational-concept book: inference-engineering chapter_ref: "Ch. 5, sec. 5.2 (pp. 129-131)" status: published last_reviewed: 2026-07-07 canonical_url: https://yourwildcard.ai/blog/how-speculative-decoding-works --- # How does speculative decoding generate more than one token per forward pass? Decode is limited by memory bandwidth, which leaves compute sitting idle. Speculative decoding uses that idle compute. A small speculator drafts several tokens ahead, and the target model validates all of them in one forward pass. Kiely compares validation to checking a finished sudoku instead of solving one. The catch is that speculation improves tokens per second and inter-token latency only, never time to first token, and only when spare compute exists. ## Understand why validation is cheaper than generation During decode, the model generates one token per forward pass, and each pass reads every weight from memory. At low batch sizes the GPU spends most of each pass waiting on those reads, so the compute units sit idle. Kiely's Inference Engineering (Ch. 5, sec. 5.2) points out that this idle compute is the resource speculation spends. The mechanism has three steps: 1. A speculator generates one or more draft tokens. 2. The target model runs one forward pass over all the drafts at once and checks whether each one matches what it would have generated. 3. The target model accepts the valid prefix of the drafts and generates one more token itself. If N drafts are accepted, the pass produces N + 1 tokens instead of 1. Checking a batch of drafts in one pass costs about the same memory traffic as generating a single token, because the weights are read once either way. The extra work is compute, and compute was idle. Kiely's sudoku comparison captures the asymmetry. Solving the puzzle is hard, and checking a finished one is easy. Generating a token is the solve, and validating a draft is the check. ## Work the arithmetic on acceptance rate Kiely lists three factors that set the speedup: draft token cost, draft sequence length, and token acceptance rate. He also notes that acceptance is high early in the draft sequence and drops the deeper you go, and that one rejection throws away every draft after it. Here is a worked example. These numbers are illustrative, not benchmarks. Suppose each draft token is accepted with probability 0.7, drafts after a rejection are discarded, and one draft token costs 10 percent of a target forward pass. At draft length 5, the expected number of accepted drafts is 0.7 + 0.49 + 0.34 + 0.24 + 0.17, which is about 1.94. Add the token the target generates itself and each pass yields about 2.94 tokens. The pass costs 1.0 for the target plus 0.5 for five drafts, so 1.5 units of time. That is about 2.94 / 1.5, or roughly 2.0 times the tokens per unit time. At draft length 10, the expected accepted count only rises to about 2.27, because tokens 6 through 10 each survive only if all five before them did. Each pass yields about 3.27 tokens but now costs 2.0 units. That is about 1.6 times baseline, worse than draft length 5. This is why Kiely tells you to aim for short draft sequences with a high acceptance percentage. Past some length, every added draft costs full price and is almost never accepted. ## Know what speculation cannot fix Kiely is blunt on the limits: "Speculative decoding only improves TPS/ITL, not TTFT." Prefill is already compute bound, so there is no idle compute to spend, and the first token comes out no sooner. The same logic caps speculation at high batch sizes. When many requests share the GPU, decode itself becomes compute bound, and there are no spare cycles for validation. Kiely notes that inference engines must dynamically disable speculation as batch size grows. Acceptance rate also moves with sampling settings. Higher temperature makes the target's choices harder to predict, so more drafts are rejected. Draft-target speculation, the variant in this post's exercise, uses a separate small model as the speculator. Kiely's rule of thumb is that the draft model should be at least ten times smaller than the target by parameter count, and ideally from the same family so the tokenizers match. ## Try it Run draft-target speculation in llama.cpp with a 0.5B draft model and a 7B target from the same family, e.g., Qwen2.5 0.5B drafting for Qwen2.5 7B. Recent llama.cpp builds use `-md` for the draft model and `--draft-max` for the draft length. Older builds call the length flag `--draft`. ```bash # Same prompt at three draft lengths for n in 2 5 10; do llama-cli \ -m qwen2.5-7b-instruct-q4_k_m.gguf \ -md qwen2.5-0.5b-instruct-q4_k_m.gguf \ --draft-max $n \ -n 256 --temp 0 \ -p "Write a step-by-step explanation of how a hash map handles collisions." \ 2>&1 | grep -Ei "drafted|accept|eval time|tokens per second" done ``` llama.cpp prints the number of drafted and accepted tokens, so you can compute acceptance rate as accepted divided by drafted. Record tokens per second and acceptance rate at each draft length, then chart tokens per second against acceptance rate. Also run once with no draft model to get your baseline. Use temperature 0 first, then repeat at 0.8 and watch the acceptance rate fall. Both models must fit in memory at once. If the 7B target already fills your GPU, the draft model spills and the run slows down for a reason that has nothing to do with speculation. ## Check yourself 1. What was your acceptance rate at draft lengths 2, 5, and 10? Expected answer: three numbers you read from your own runs, and the rate should fall as the draft gets longer. 2. At which draft length did tokens per second peak, and why did longer drafts stop paying? Expected answer: the peak is usually at a short length, because late draft tokens are rarely accepted but still cost a full draft pass each. 3. Did speculation change your time to first token? Expected answer: no. Prefill is compute bound, so speculation has no idle compute to use there. 4. What happened to acceptance rate when you raised temperature to 0.8? Expected answer: it dropped, because the target's sampled choices became harder for the draft model to predict. ## Next steps - [Speculative decoding guide](https://yourwildcard.ai/docs/technical-guides/speculative-decoding.md) for the algorithm variants past draft-target, such as EAGLE and n-gram speculation. - [llama.cpp in the ecosystem](https://yourwildcard.ai/docs/ecosystem/llama-cpp.md) for setup, quantization choices, and the rest of the local toolchain. When you can do this, you can configure draft-model speculation locally and identify the acceptance-rate regime where it is net-positive. --- --- title: "How do I benchmark an LLM server properly?" description: "A good benchmark mirrors production on sequence lengths, traffic, request contents, and sampling settings, and changes one variable at a time." audience: inference-engineer pillar: ecosystem-player book: inference-engineering chapter_ref: "Ch. 4, secs. 4.5-4.5.2 Benchmarking" status: published last_reviewed: 2026-07-07 canonical_url: https://yourwildcard.ai/blog/how-to-benchmark-an-llm-server --- # How do I benchmark an LLM server properly? A good benchmark mirrors production on four axes: sequence lengths, traffic volume and pattern, request contents, and sampling parameters. The best way to get there is to shadow real traffic onto the test system. Then measure a baseline and change one variable at a time. If you tune against unrealistic inputs, the gains disappear in production, and some optimizations conflict, e.g., speculative decoding works against large batches. ## Mirror production on four axes Kiely's Inference Engineering (Ch. 4, sec. 4.5) states the goal directly: "A high-quality benchmark simulates real life as closely as possible." The best version of this is shadowing. Shadowing means you copy incoming production requests onto the test system, so you can measure the test system without touching the original request. If you cannot shadow real usage, you have to simulate it, and the book lists four dimensions your simulated traffic has to match. - **Sequence lengths.** Time to first token and memory use depend on the input sequence length and the output sequence length, which are the token counts of the prompt and the response. A benchmark with 100-token prompts says nothing about a product where users paste 8,000-token documents. - **Volume and pattern of traffic.** Batching and server load depend on how many requests arrive at once. The book also says to jitter the traffic, which means adding random variation to request timing so it looks like real usage instead of a fixed drumbeat. - **Request contents.** The actual text of each request changes results through factors like the cache hit rate and, for speculative decoding, the draft token acceptance rate. Sending the same prompt 1,000 times gives you a cache hit rate no real workload will ever have. - **Sampling parameters.** Settings like temperature and reasoning effort change how inference runs, so set them to the values production will use. Kiely's summary of why this list exists is blunt. If you maximize benchmark performance against bad inputs, performance in production will not match your expectations. The numbers you publish to your team will be true for a workload that does not exist. For request contents, the book suggests open eval datasets, e.g., MMLU for general prompts or SWE-bench for coding prompts. These datasets give you varied, realistic inputs, and you can also spot check that a performance change did not hurt output quality. Pick the dataset that matches your product, e.g., HumanEval when you are tuning a code completion system. ## Start from a baseline and change one thing Kiely (sec. 4.5.2) adds a second requirement on top of realism. A benchmark also has to be consistent. Send enough traffic that one slow outlier cannot sway the result, and when in doubt run the benchmark several times and average. Before any optimization work, record a solid baseline at a fixed configuration. Then test each optimization on its own, and also together, so you know which change actually drives the improvement. The book warns that in some cases "optimizations can work against each other", and its example is running speculative decoding with large batch sizes. Speculative decoding spends spare compute to guess tokens ahead, and large batches consume that same spare compute, so stacking both can be worse than either alone. The same discipline applies to the benchmark setup itself. You will want to test different traffic patterns and sequence shapes, but change only one variable per comparison, or you cannot say which change caused the result. ## Work one comparison with noise bounds Here is a made-up but arithmetically honest example. Suppose you run your baseline three times at batch size 8, and the P90 TTFT comes back as 402 ms, 418 ms, and 410 ms. P90 TTFT is the time to first token that 90 percent of requests beat. The spread across the three runs is 16 ms around a mean of 410 ms. That 16 ms is your noise bound. Any change smaller than that is indistinguishable from run-to-run noise. Now change exactly one flag, batch size 8 to 16, and rerun. | Metric | Batch size 8 (baseline) | Batch size 16 | |---|---|---| | P90 TTFT | 410 ms (402 to 418 across runs) | 455 ms | | Total tokens/sec | 240 | 380 | The TTFT moved 45 ms, which is about three times the 16 ms noise bound, so the latency regression is real. The throughput gain of 140 tokens/sec is also real. Neither number is good or bad on its own. The comparison tells you the price of the throughput, and your latency target tells you whether to pay it. If you had changed batch size and quantization level in the same run, this table would be unreadable. You could not assign the 45 ms to either change. One variable per comparison is what makes the table mean something. ## Try it Time budget is about 30 minutes. You need a local model server running, e.g., llama.cpp's server on a small model. 1. Pick fixed settings and write them down: model, quant level, batch size, prompt set, temperature, and concurrency. 2. Run a baseline three times. With llama.cpp you can use its built-in tool for a quick single-machine read: ```bash llama-bench -m model.gguf -p 512 -n 128 ``` For concurrent traffic against a running server, use Locust with a small script that posts your real prompt set: ```bash pip install locust locust -f llm_bench.py --headless -u 8 -r 2 -t 3m --host http://localhost:8080 ``` SGLang's genai-bench is another option built for exactly this job. Record P50 and P90 TTFT and tokens/sec for each run, and note the spread across the three runs as your noise bound. 3. Change exactly one flag. Either double the batch size or drop one quant level, not both. 4. Rerun the same benchmark and put the before and after numbers in one table next to your noise bound. ## Check yourself - **Which four axes does your benchmark match against production?** Expected answer: sequence lengths, traffic volume and pattern, request contents, and sampling parameters. If you cannot say how your prompt set matches production lengths and contents, the benchmark is not mirroring anything. - **What is your noise bound, and how did you get it?** Expected answer: a number like "16 ms on P90 TTFT", taken from repeated baseline runs at identical settings, not guessed. - **In your before and after table, how many variables changed?** Expected answer: exactly one, and you can name it. If two changed, the comparison is invalid and you rerun it. - **Is your measured change bigger than your noise bound?** Expected answer: yes by a clear multiple, or you treat the result as no change. ## Next steps - [Benchmarking guide](https://yourwildcard.ai/docs/technical-guides/benchmarking.md) covers tool setup in more depth. - [llama.cpp](https://yourwildcard.ai/docs/ecosystem/llama-cpp.md) documents the local server this exercise runs against. - [What do TTFT and TPS actually measure?](/blog/what-do-ttft-and-tps-measure) defines the metrics you just recorded, and why per-user and system tokens/sec are different numbers. When you can do this, you can design a benchmark that mirrors production and run a valid comparison where only one variable changed, with known noise bounds. --- --- title: "Why do images blow up my LLM's context window?" description: "A high-resolution image adds roughly a thousand visual tokens, so VLM optimization is mostly a long-context and KV cache problem. LLM techniques transfer." audience: inference-engineer pillar: foundational-concept book: inference-engineering chapter_ref: "Ch. 6, sec. 6.1 (pp. 156-158)" status: published last_reviewed: 2026-07-07 canonical_url: https://yourwildcard.ai/blog/images-blow-up-context-window --- # Why do images blow up my LLM's context window? A single high-resolution image adds roughly a thousand visual tokens to the input sequence. So most vision language model optimization is a long-context and KV cache problem. Once you treat images as bulk tokens, your LLM playbook transfers directly, e.g., a quantized KV cache works the same way on image tokens as on text tokens. ## See where the tokens come from A vision language model is two modules. One is a standard large language model. The other is a small vision encoder that cuts an image into patches and converts each patch into an image token. The encoder is tiny next to the language model. Kiely's Inference Engineering (Ch. 6, sec. 6.1) gives Mistral Large 3 as an example, where the vision encoder is two billion parameters next to a 673B language model. After encoding, the image tokens enter the input sequence next to your text tokens and flow through prefill and decode like any other tokens. Kiely's rule of thumb is that a high-resolution image "adds about a thousand visual tokens to the input sequence" (Ch. 6, sec. 6.1). A typical chat turn is a few hundred text tokens, so one image can outweigh all the text in the request. The context window does not care that the tokens came from pixels. Every image token takes a sequence position, gets attended to on every decode step, and holds a slot in the KV cache until the request finishes. ## Count what one image costs Here is a worked example. The model config is an example, not a benchmark. Take an 8B language model with 32 layers, 8 KV heads, a head dimension of 128, and an FP16 KV cache. Each token stores a key and a value in every layer: - Bytes per token = 2 (key and value) x 32 layers x 8 heads x 128 dims x 2 bytes = 131,072 bytes, which is 128 KB. - One high-resolution image at about 1,000 tokens = about 128 MB of KV cache. - A request with 4 images and 200 text tokens = about 4,200 input tokens and about 525 MB of KV cache, for one request. Prefill cost grows the same way. The 4,200-token request runs prefill on 21 times more tokens than the 200-token text version of the same question, so time to first token grows with it. This is why a chat app that feels fast on text can stall the moment a user attaches screenshots. Video is the extreme case. Kiely works the numbers in sec. 6.1.1: cinematic video is 24 frames per second, so at about 1,000 tokens per frame, a four-second clip approaches 100,000 input tokens. No one serves that raw, which is why the book calls downsampling practically obligatory for video. ## Reuse your LLM playbook Kiely's point in sec. 6.1 is that the primary VLM optimization challenge is the longer input sequence and the larger KV cache, and that every technique from the long-context chapter applies: - [KV cache quantization](https://yourwildcard.ai/docs/technical-guides/kv-cache.md) cuts the memory that those thousand tokens per image hold, e.g., FP8 halves the 128 MB above. - [Prefix caching](https://yourwildcard.ai/docs/technical-guides/prefix-caching.md) reuses the stored keys and values for an image across turns, so a multi-turn chat about one screenshot pays its prefill once. - Disaggregated prefill moves the image-heavy prefill work to separate workers that scale on their own, so long image requests do not stall decode for everyone else. One thing differs from text serving. Vision encoders vary a lot across models, so runtime support is more fragmented than for plain LLMs. Check that your serving framework supports your exact model before you plan around it. vLLM and SGLang have the widest VLM coverage. ## Pick a resolution on purpose VLMs add one knob that text serving does not have: input resolution. Kiely notes that a high-resolution image takes about four times more tokens than a low-resolution one, and provides more detail in return. That is the whole trade. More pixels means more tokens, more KV cache, and slower prefill, in exchange for answers that can read finer detail. The default in most client libraries is to send the image at whatever size the user uploaded. That default is a policy choice someone else made for you. The exercise below finds the resolution where your task's answer quality stops improving, and that resolution is your downsampling target. ## Try it Send the same image to a local VLM at three resolutions and log token count, time to first token, and answer quality. This takes under 30 minutes. No GPU is needed. A 7B VLM at 4-bit needs about 6 GB of memory and runs on a laptop, and prefill is slower on CPU than on a GPU, which makes the resolution effect easier to see. Start a server with a vision model that llama.cpp supports: ```bash brew install llama.cpp llama-server -hf ggml-org/Qwen2.5-VL-7B-Instruct-GGUF --port 8080 ``` Pick a test image with fine detail, e.g., a dense dashboard screenshot, and a question whose answer sits in that detail. Then run each resolution: ```python import base64, io, time, requests from PIL import Image img = Image.open("test.png") for width in [336, 672, 1024]: scaled = img.resize((width, int(img.height * width / img.width))) buf = io.BytesIO(); scaled.save(buf, format="PNG") uri = "data:image/png;base64," + base64.b64encode(buf.getvalue()).decode() start = time.time() r = requests.post("http://localhost:8080/v1/chat/completions", json={ "messages": [{"role": "user", "content": [ {"type": "image_url", "image_url": {"url": uri}}, {"type": "text", "text": "What is the exact value in the top right panel?"}]}]}) body = r.json() print(width, body["usage"]["prompt_tokens"], f"{time.time() - start:.1f}s", body["choices"][0]["message"]["content"][:60]) ``` The script prints total latency rather than time to first token. Add `"stream": True` and time the first chunk if you want TTFT itself. Fill a table with resolution, prompt tokens, latency, and whether the answer was right. You should see the token count climb steeply with resolution. The check is whether the answer got better along with it. Find the row where quality stopped improving but tokens kept climbing. That resolution is your downsampling target, and one line of resize code before the API call enforces it. ## Check yourself 1. A user sends a request with 3 high-resolution images and 150 words of text. Roughly what fraction of the input tokens are image tokens? Expected answer: about 3,000 image tokens against about 200 text tokens, so image tokens are roughly 94 percent of the input. 2. Your VLM server runs out of KV cache memory under image-heavy load. Name two fixes from the LLM playbook that apply unchanged. Expected answer: quantize the KV cache to cut bytes per token, and use prefix caching so repeated images in multi-turn chats do not hold duplicate entries. Serving at a lower input resolution also works and is the VLM-specific fix. 3. Why does a four-second video clip threaten to produce nearly 100,000 input tokens? Expected answer: cinematic video is 24 frames per second, each frame is an image at about 1,000 tokens, and 4 x 24 x 1,000 is 96,000, so downsampling the resolution and frame rate is required in practice. ## Next steps - Read the [KV cache guide](https://yourwildcard.ai/docs/technical-guides/kv-cache.md) to size and quantize the cache that image tokens fill. - Read the [prefix caching guide](https://yourwildcard.ai/docs/technical-guides/prefix-caching.md) to reuse image prefill across turns. - For why prefill is the phase that image tokens hurt most, read [What are prefill and decode?](/blog/what-are-prefill-and-decode) from this series. When you can do this, you can quantify the token and latency cost of image resolution choices and set a downsampling policy. --- --- title: "Inference is fast but my app feels slow — what do I check?" description: "Track on-GPU inference time and end-to-end time. When inference is fast but end-to-end is slow, the fix is in infrastructure, not the model." audience: inference-engineer pillar: foundational-concept book: inference-engineering chapter_ref: "Ch. 1, sec. 1.4.2 (p. 37)" status: published last_reviewed: 2026-07-07 canonical_url: https://yourwildcard.ai/blog/inference-fast-but-app-feels-slow --- # Inference is fast but my app feels slow — what do I check? Track two numbers. The first is on-GPU inference time, and the second is end-to-end time including network and queueing. When inference is fast but end-to-end is slow, the fix is in infrastructure such as proxies, queues, and cold replicas, not in the model. This one comparison tells you where to spend your optimization effort. ## Measure the two numbers separately End-to-end latency is the time from the moment your app sends a request until the full response arrives. Inference time is only the part the model spends generating tokens on the GPU. Everything else is overhead. The overhead includes network transit, TLS handshakes, time spent waiting in a queue, and time spent starting a replica that was scaled to zero. Kiely's Inference Engineering (Ch. 1, sec. 1.4.2) makes this the second key distinction in latency metrics, after percentiles. The book defines inference time as the on-GPU time required to generate tokens, and end-to-end time as the measurement that also counts network latency and queue time. Both are useful. Inference time tells you whether your model performance work is paying off. End-to-end time tells you what your users actually experience. Most teams only look at one of the two. A dashboard that only shows server-side inference time can look healthy while users wait three seconds for every answer. A dashboard that only shows end-to-end time cannot tell you whether the model or the infrastructure is slow. ## Compute the overhead and route the fix Once you have both numbers for the same request, subtract them. The difference is your infrastructure overhead. Kiely gives the routing rule directly. When inference is fast but end-to-end is slow, "turn your attention to infrastructure rather than model performance optimization". Here is a worked example. The numbers are hypothetical, but the arithmetic is what you would do with your own logs. - The server logs say a request spent 0.12 seconds in prefill and 1.60 seconds in decode, so inference time is 1.72 seconds. - The client measured 2.90 seconds from send to last byte. - The overhead is 2.90 minus 1.72, which is 1.18 seconds. - The overhead share is 1.18 divided by 2.90, which is about 41 percent of end-to-end latency. At 41 percent, quantizing the model or tuning the batch size cannot fix most of what the user feels. Even if you cut inference time in half, end-to-end time only drops from 2.90 to about 2.04 seconds. The bigger win is in the 1.18 seconds of overhead. Common suspects, roughly in the order to check them: - **Queue wait.** The request sat in a queue because every replica was busy. Look for a queue depth or scheduler wait metric on your serving layer. - **Cold replicas.** The platform scaled to zero and had to load model weights before serving. Cold starts show up as a slow P99 while the P50 stays fine, which is one reason to [report percentiles, not averages](/blog/why-report-p99-latency-not-average). - **Proxies and gateways.** Each hop adds time, and a proxy that buffers the response instead of streaming it hides your fast first token entirely. - **Network transit.** Distance and TLS setup add fixed cost per request. This one you can bound with a simple ping or a TLS timing tool. If instead the overhead is small, e.g., 5 percent, the model is your bottleneck and you should work on inference itself. The [inference stack](https://yourwildcard.ai/docs/concepts/inference-stack.md) doc breaks down which layer owns which kind of fix. ## Try it The exercise is to add two timestamps to one request path and compute the overhead as their difference. It takes under 30 minutes and needs no GPU, because llama.cpp runs on a CPU or on Apple silicon. Start a local server with [Bonsai 8B](https://huggingface.co/prism-ml/Bonsai-8B-gguf), PrismML's open-weight model, using the [llama.cpp recipe](https://yourwildcard.ai/docs/build-and-run/bonsai-llamacpp.md): ```bash llama-server -m Bonsai-8B-Q1_0.gguf --port 8080 ``` llama.cpp reports its own server-side timings in every response, so the server timestamp is free. Save this as `overhead.py` and run it with `python3 overhead.py`. ```python import time, requests URL = "http://127.0.0.1:8080/completion" payload = {"prompt": "Explain what a proxy does in two sentences.", "n_predict": 150} start = time.perf_counter() r = requests.post(URL, json=payload).json() total_ms = (time.perf_counter() - start) * 1000 t = r["timings"] inference_ms = t["prompt_ms"] + t["predicted_ms"] overhead_ms = total_ms - inference_ms print(f"end-to-end: {total_ms:.0f} ms") print(f"inference: {inference_ms:.0f} ms") print(f"overhead: {overhead_ms:.0f} ms " f"({overhead_ms / total_ms * 100:.0f}% of end-to-end)") ``` On localhost the overhead should be small, since there is no real network and no queue. Now point `URL` at your production endpoint, or add a hop such as an nginx proxy in front of the local server, and run it again. The overhead number grows and the inference number does not. That gap is the thing this post is about. If your production server does not return timings in the response, log the inference duration server side and join the two records by request ID. The [observability guide](https://yourwildcard.ai/docs/technical-guides/observability.md) shows how to set that up. ## Check yourself 1. Your model dashboard shows P50 inference time of 800 ms, but users report two-second waits. What number do you need before you change anything? Expected answer: the client-side end-to-end time for the same requests. Without it you cannot tell whether the missing 1,200 ms is infrastructure overhead or a measurement gap. 2. End-to-end P50 is 3.0 seconds and server-side inference is 2.8 seconds. Should you tune your load balancer? Expected answer: no. Overhead is only about 7 percent, so the model is the bottleneck and the fix is inference work such as a smaller model or quantization. 3. P50 end-to-end looks fine but P99 is ten times worse, while inference time is stable at both percentiles. What is a likely suspect? Expected answer: queueing or cold starts. Both add large, occasional delays outside the model, which inflates the tail without moving inference time. ## Next steps - [The inference stack](https://yourwildcard.ai/docs/concepts/inference-stack.md) maps which layer, from hardware to gateway, owns each kind of latency. - [Observability](https://yourwildcard.ai/docs/technical-guides/observability.md) covers logging inference time server side and joining it with client-side traces. - [Why report P99 latency, not average](/blog/why-report-p99-latency-not-average) explains the percentile half of Kiely's chapter 1 metrics advice. When you can do this, you can decompose end-to-end latency into model and infrastructure components and route the fix to the right layer. --- --- title: "Is a bigger model ever cheaper? Rethinking cost per unit of capability" description: "Occasionally yes. When a bigger model's pass rate removes retries or human review, its cost per successful task can undercut a cheap model that fails often." audience: ml-product-team pillar: end-user-case-study book: transcript-theme chapter_ref: "Theme 1: Intelligence density (capability per GB/watt/dollar)" status: published last_reviewed: 2026-07-07 canonical_url: https://yourwildcard.ai/blog/is-a-bigger-model-ever-cheaper --- # Is a bigger model ever cheaper? Rethinking cost per unit of capability Occasionally yes. When a bigger model's pass rate removes retries, human review, or a second pipeline stage, its cost per successful task can undercut a cheaper model that fails often. The right unit is cost per unit of capability delivered, not cost per token. Quantized mid-size models that fit your hardware win this math surprisingly often. ## Price the success, not the token A per-token price tells you what one attempt costs. It does not tell you what one finished task costs, because a failed attempt still bills you for its tokens and then bills you again for whatever cleans up the failure. If the small model fails 30 percent of the time and the big model fails 2 percent of the time, the per-token comparison hides the largest cost in the pipeline. This is the dollar denominator of intelligence density, the theme this post draws on, which scores a model by capability per GB, per watt, and per dollar. The numerator is capability delivered, meaning tasks that pass your success check. A cheap attempt that fails delivers zero capability, so it makes the density worse, not better. The [intelligence density concept page](https://yourwildcard.ai/docs/concepts/intelligence-density.md) covers the other two denominators. ## Add retries and fallbacks to the formula Cost per successful task has three parts. - The cost of one attempt, which is tokens per attempt times the price per token. - The expected number of attempts, which grows as the pass rate drops. With a retry cap of three, the expected attempt count is 1 plus the failure rate plus the failure rate squared. - The fallback cost, which is what you pay when every retry fails, e.g., a human reviews the item. Multiply that cost by the probability that all attempts fail. So the formula is: ``` cost per task = (cost per attempt x expected attempts) + (P(all attempts fail) x fallback cost) ``` The fallback term is where small models lose. Human review at even a few dollars per item costs far more than any saving on tokens, because token costs are fractions of a cent and people are not. ## Work the numbers for invoice extraction Here is the arithmetic for a made-up but realistic pipeline. Every number is an input to the formula, not a benchmark claim, so rerun it with your own measurements. The task is extracting fields from an invoice into JSON. Success means the output validates against your schema and passes your field checks. A failed item retries up to three times, and if all three attempts fail, a person fixes it at an estimated 2 dollars of labor per item. - Model S is small and cheap. Suppose one attempt costs $0.0004 and the pass rate on your eval set is 0.70. - Model L is large and 10x more expensive per token. Suppose one attempt costs $0.004 and the pass rate is 0.98. For Model S, the expected attempt count is 1 + 0.30 + 0.09, which is 1.39, so attempts cost 1.39 times $0.0004, which is about $0.00056. The probability that all three attempts fail is 0.30 cubed, which is 0.027. The fallback term is 0.027 times 2 dollars, which is $0.054. Total: about $0.055 per task. For Model L, the expected attempt count is 1 + 0.02 + 0.0004, which is about 1.02, so attempts cost about $0.0041. The probability that all three attempts fail is 0.02 cubed, which is 0.000008, so the fallback term is about $0.00002. Total: about $0.0041 per task. Model S is 10x cheaper per token and about 13x more expensive per finished task, because 2.7 percent of its items fall through to a person. The per-token ranking and the per-task ranking point in opposite directions. ## Know when the mid-size model wins anyway The example above compares two API prices. A third option often beats both. A quantized mid-size model, e.g., [Bonsai 8B in GGUF format](https://huggingface.co/prism-ml/Bonsai-8B-gguf), fits on hardware you already own, so its marginal cost per attempt is close to zero. If its pass rate on your eval is close to the big model's, its cost per successful task is almost entirely the fallback term, and it wins the density math outright. That is why fit-to-hardware models win this comparison so often. The [cost modeling guide](https://yourwildcard.ai/docs/technical-guides/cost-modeling.md) shows how to spread hardware cost over its useful life so the comparison stays honest. The pass rates above are inputs you must measure, not properties of model size. A bigger model is not always more accurate on your task, and the formula only rewards it when it is. ## Try it Compute cost per successful task for a small and a large model on one task. Budget 30 minutes. This runs on a laptop CPU, no GPU needed, though the 8B model needs about 5 GB of free RAM. First, write 10 inputs for a task with a checkable success rule, e.g., extraction that must validate against a JSON schema. Decide the pass rule before you run anything. ```bash # Install llama.cpp (macOS) brew install llama.cpp # Download a small and a mid-size instruct model in 4-bit GGUF # format from Hugging Face, then run your inputs through each: while IFS= read -r p; do llama-cli -m model-small-q4_k_m.gguf -p "$p" -n 256 --temp 0 -no-cnv done < inputs.txt while IFS= read -r p; do llama-cli -m model-8b-q4_k_m.gguf -p "$p" -n 256 --temp 0 -no-cnv done < inputs.txt ``` Count the passes for each model. For local runs, use each model's published API price as the cost per attempt so the comparison is fair, or use your amortized hardware cost. Then apply the formula: expected attempts is 1 plus failure rate plus failure rate squared, and the fallback term is the failure rate cubed times your honest estimate of cleanup cost. Compare the two totals against the two per-token prices. ## Check yourself 1. Do your two rankings, cost per token and cost per successful task, disagree? If they agree, can you explain why, e.g., both models pass nearly everything, so retries and fallbacks never fire? 2. In the invoice example, why does Model S lose despite a 10x price advantage? Expected answer: 2.7 percent of its items reach the 2 dollar human fallback, and the $0.054 fallback cost is far larger than the $0.003 saved on tokens. 3. What single measured number would flip your result? Expected answer: the small model's pass rate. If it rises enough that the fallback term shrinks below the token saving, the small model wins again. ## Next steps - [Intelligence density](https://yourwildcard.ai/docs/concepts/intelligence-density.md) explains capability per GB, per watt, and per dollar, of which this post worked the dollar case. - [Cost modeling](https://yourwildcard.ai/docs/technical-guides/cost-modeling.md) covers amortizing hardware and pricing retries in production pipelines. - [What does 'intelligence density' actually measure?](/blog/what-is-intelligence-density) works the GB denominator the same way this post works dollars. When you can do this, you can compute cost per successful task including retries and use it to challenge a model comparison that is based on price per token. --- --- title: "Is generative AI the same thing as AI?" description: "No. GenAI is one of four groups of AI, alongside traditional AI, general intelligence, and superintelligence. Conflating them wastes money and erodes trust." audience: ml-product-team pillar: foundational-concept book: building-ai-products chapter_ref: "Nika, Ch. 1, 'The Stages of AI Evolution' and Figure 1-1" status: published last_reviewed: 2026-07-07 canonical_url: https://yourwildcard.ai/blog/is-generative-ai-the-same-as-ai --- # Is generative AI the same thing as AI? No. Generative AI is one of four groups of AI. Marily Nika's Building AI-Powered Products (Ch. 1, Figure 1-1) splits modern AI into traditional AI, generative AI, general intelligence, and superintelligence, and she calls the mixup of "AI" with GenAI a misconception she meets daily. The mixup has costs. Picking an LLM where a classifier suffices wastes latency and money, and users stop trusting marketing that implies general intelligence. ## Learn the four groups Nika writes that GenAI "is by no means a replacement for traditional AI" (Building AI-Powered Products, Ch. 1). Her Figure 1-1 draws the four groups as rings of widening scope. - Traditional AI, 1950s to present. Systems built for one specific task, using rules or pattern recognition. Examples in the chapter include face recognition in photo apps, speech-to-text in voice assistants, language translation, and pattern recognition over large datasets. - Generative AI, late 2010s to present. Systems that create content such as text, images, video, or music from a prompt. Examples in the chapter include chat models, DALL-E, and procedurally generated game worlds. - Artificial general intelligence, which Nika dates as a question mark around the 2030s. A hypothetical system that could learn and apply knowledge across many domains the way a person can. It does not exist yet. - Artificial superintelligence, which she dates as a question mark around the 2040s. A hypothetical system beyond human intelligence. It also does not exist. Only the first two groups ship today. When someone says "we added AI" they are describing something in ring one or ring two, and the useful question is which one. The recommendation feed in a music app is traditional AI. The feature that writes a playlist description is generative AI. Same app, different rings, different engineering. ## Work the arithmetic on picking the wrong ring Calling everything "AI" leads to a concrete engineering mistake. When GenAI is the only ring you can name, every problem looks like a prompt, so teams route traditional AI problems through an LLM. The compute arithmetic shows what that costs. Take support ticket routing into six queues. This is single-task classification, a ring one problem. - Option A is a logistic regression classifier on a 50,000 word vocabulary. Scoring one ticket takes about 50,000 multiply-add operations per class, so about 300,000 operations for six classes. - Option B is an 8B parameter LLM such as [Bonsai 8B](https://huggingface.co/prism-ml/Bonsai-8B-gguf). A transformer does roughly 2 operations per parameter per token, so about 16 billion operations for each token it processes. A 300 token ticket costs 300 times that in prefill, about 4.8 trillion operations, before the first label token comes out. Divide the two. 4.8 trillion over 300,000 is 16 million. The LLM spends about 16 million times the compute of the classifier on the same ticket, and this rough count ignores attention overhead, so the true ratio is higher. That compute gap is where the latency and the bill come from, and the LLM also adds a failure mode the classifier does not have, because it can answer with text that is not one of your six labels. The LLM is still the right pick in some cases, e.g., when you have no labeled tickets yet and need working routing this week. That is a reasoned choice between rings. The category error is not knowing a choice existed. The [intelligence density](https://yourwildcard.ai/docs/concepts/intelligence-density.md) page covers how to think about capability per unit of compute when you make this call. The estimate above is derived from parameter counts, not measured. Measured throughput depends on hardware, batch size, and quantization. The point survives any reasonable measurement, because seven orders of magnitude do not disappear into implementation details. ## Check the message against the ring The same mixup damages trust when it runs through marketing instead of tooling. Nika's Figure 1-1 puts general intelligence in a ring that does not exist yet, with a question mark on the date. Copy that says an assistant "understands your business" or "thinks like your best analyst" describes ring three while the product sits in ring one or two. Users find the gap fast, e.g., the first time the assistant fails a task that any junior analyst would handle, and the copy has taught them to distrust every other claim you make. The fix is to write copy that names what the system does in its actual ring. "Drafts a reply you review" is a ring two claim a ring two product can keep. This is the self check for your own project: the ring your product sits in matches what your copy implies, or you can name the gap out loud. ## Try it This takes about 25 minutes. The optional command runs on a laptop CPU or Apple silicon, no GPU needed, with a several GB model download. 1. Take 10 minutes and list five AI features you used today, e.g., a spam filter, face unlock, a photo search, an autocomplete, a chat assistant. 2. Label each one traditional or generative, with one sentence of justification tied to the definition. Traditional means one specific task through rules or pattern recognition. Generative means it created content from a prompt. 3. Sketch the four rings on paper: traditional, generative, general intelligence, superintelligence. Place your current project on it, and write one sentence saying why it sits there. 4. Optional. Watch a ring two tool do a ring one job. Run an LLM as a ticket classifier: ```bash brew install llama.cpp llama-cli -hf prism-ml/Bonsai-8B-gguf \ -p "Classify this ticket as billing, bug, or feature-request. Ticket: my invoice charged me twice this month. Label:" -n 4 ``` Time it. Then note that a keyword rule matching "invoice" or "charged" gives the same label in microseconds. The label is correct either way. The cost is not. ## Check yourself - **A bank flags unusual card transactions. Which ring?** Expected answer: traditional AI. It is one specific task done with pattern recognition, and nothing is generated. - **A coding assistant writes a unit test from a comment. Which ring?** Expected answer: generative AI. It creates content from a prompt, and it is still specialized, not general intelligence. - **A teammate says "let's use AI to find duplicate records" and means an LLM. What do you ask?** Expected answer: whether this is a classification task with available labels, because if it is, a ring one model does it at a tiny fraction of the compute, and the LLM should be a reasoned choice, not a default. - **Does your product's ring match your marketing copy?** Expected answer: yes, or you can name the exact claim that overshoots and which ring it borrows from. ## Next steps - Read the [inference stack](https://yourwildcard.ai/docs/concepts/inference-stack.md) page to see where each kind of model runs and what serving it costs. - Read the [intelligence density](https://yourwildcard.ai/docs/concepts/intelligence-density.md) page for how to compare capability per unit of compute across model choices. - For the same tradeoff argued from the serving side, read [why model selection is the biggest inference optimization](/blog/model-selection-biggest-inference-optimization). When you can do this, you can classify AI features into the four groups and catch category errors in tooling choices and in messaging. --- --- title: "How does the KV cache make attention linear instead of quadratic?" description: "The KV cache stores key/value pairs for prior tokens so each decode step computes attention only for the new token. The cost is GPU memory." audience: inference-engineer pillar: foundational-concept book: inference-engineering chapter_ref: "Ch. 2, sec. 2.2.3 (pp. 52-53)" status: published last_reviewed: 2026-07-07 canonical_url: https://yourwildcard.ai/blog/kv-cache-linear-attention --- # How does the KV cache make attention linear instead of quadratic? Attention is quadratic in sequence length, but the KV cache stores key/value pairs for every prior token so each decode step only computes attention for the new token, which is linear time. The cost is GPU memory. KV cache size is the hidden budget that caps your batch size and context length. ## Understand why attention starts out quadratic Attention relates each token to every token before it. To produce token number n, the model compares that token's query against n prior keys. If you recomputed everything from scratch at every step, generating a sequence of length n would take on the order of n squared comparisons in total, because step 1 does 1 comparison, step 2 does 2, and so on up to n. Kiely's Inference Engineering (Ch. 2, sec. 2.2.3) states the problem plainly. Because attention checks the current token against every previous token, it is "a quadratic-time equation with respect to sequence length". As context grows, attention slows. ## See what the cache removes The keys and values for a token never change once the model has computed them. Token 5's key is the same whether you are generating token 6 or token 6,000. So there is no reason to recompute it. The KV cache stores the key and value vectors for every token the model has already processed, in every layer and every attention head. During prefill the model computes keys and values for the whole prompt and writes them into the cache. During each decode step the model computes keys and values for one new token, appends them, and reads everything else from the cache. Each step is now one query against n stored keys, so the work per step grows linearly with context instead of the whole history being recomputed. Kiely notes that the cache lives on GPU memory by default. That is the trade. You buy linear compute by spending memory, and the memory bill grows with every token in every active request. ## Compute the cache size for an 8B model The formula is a straight multiplication: ``` bytes = layers x kv_heads x head_dim x 2 x bytes_per_value x tokens ``` The 2 is there because you store both a key and a value per head per layer. Most modern 8B models use grouped query attention, so the number of KV heads is smaller than the number of attention heads. Llama 3.1 8B has 32 query heads but only 8 KV heads. Read `num_key_value_heads` from the model's config.json, not `num_attention_heads`, or your estimate will be 4x too high. For Llama 3.1 8B, config.json gives 32 layers, 8 KV heads, and a head dimension of 128. In FP16 each value takes 2 bytes. Per token that is: ``` 32 layers x 8 kv_heads x 128 head_dim x 2 (K and V) x 2 bytes = 131,072 bytes = 128 KiB per token ``` At an 8K context (8,192 tokens): ``` 8,192 tokens x 128 KiB = 1 GiB ``` One request at 8K context holds 1 GiB of cache before you count the model weights. Eight concurrent requests at 8K hold 8 GiB. This is why the cache, not the weights, is usually what caps batch size on a fixed GPU. With a quantized cache the per value size drops. An 8 bit cache format stores about 1 byte per value, so the same 8K context takes about 0.5 GiB. A 4 bit format takes about 0.25 GiB. These quantized figures are estimates because formats like q8_0 carry a small amount of scale metadata on top of the raw values. ## Try it This takes about 20 minutes with llama.cpp and any 8B GGUF model. First, predict. Using the formula above, write down your expected cache size at 2K, 8K, and 32K context in FP16. Then measure. Run the model at each context size and read what llama.cpp allocates: ```bash llama-cli -m llama-3.1-8b-instruct-q4_k_m.gguf -c 2048 -n 8 -p "hi" 2>&1 | grep -i "KV buffer\|kv_cache" llama-cli -m llama-3.1-8b-instruct-q4_k_m.gguf -c 8192 -n 8 -p "hi" 2>&1 | grep -i "KV buffer\|kv_cache" llama-cli -m llama-3.1-8b-instruct-q4_k_m.gguf -c 32768 -n 8 -p "hi" 2>&1 | grep -i "KV buffer\|kv_cache" ``` The log reports the cache allocation in MiB. Note that llama.cpp defaults the cache to FP16 even when the weights are quantized, so your FP16 prediction is the right one to compare against. Now quantize the cache and rerun the 8K case: ```bash llama-cli -m llama-3.1-8b-instruct-q4_k_m.gguf -c 8192 -n 8 -p "hi" \ --cache-type-k q8_0 --cache-type-v q8_0 -fa on 2>&1 | grep -i "KV buffer\|kv_cache" ``` Flash attention (`-fa on`) is required to quantize the V cache. The reported size should land near half of your FP16 number. ## Check yourself 1. How many bytes does one token of KV cache take for Llama 3.1 8B in FP16? Expected answer: 128 KiB, from 32 x 8 x 128 x 2 x 2. 2. You grow context from 2K to 32K. What happens to the cache size? Expected answer: it grows 16x, linearly with tokens, from 256 MiB to 4 GiB per request. 3. Your hand calculation says 1 GiB at 8K but llama.cpp reports 1.07 GB. Is your math wrong? Expected answer: no. The log may print GB (powers of 10) while you computed GiB (powers of 2), and quantized formats add scale metadata. Anything within about 20 percent means your model of the memory is correct. 4. Why does a 70B model with the same 8 KV heads and head dimension of 128 still use much more cache per token? Expected answer: it has 80 layers instead of 32, and layers multiply directly into the formula. ## Next steps - [KV cache technical guide](https://yourwildcard.ai/docs/technical-guides/kv-cache.md) covers cache reuse and eviction in production engines. - [Prefill and decode](https://yourwildcard.ai/docs/concepts/prefill-decode.md) explains the two phases that build and consume the cache. When you can do this, you can compute KV cache memory for any model, context, and precision combination and verify it against observed usage. --- --- title: "What belongs in a local-AI runbook for a compliance-bound team?" description: "A runbook an end user can hand to auditors covers four things demos never do: rollback, model provenance, failure modes, and operational boundaries." audience: inference-engineer pillar: end-user-case-study book: transcript-theme chapter_ref: "Theme 3: How regulated end users evaluate local AI" status: published last_reviewed: 2026-07-07 canonical_url: https://yourwildcard.ai/blog/local-ai-runbook-for-compliance-teams --- # What belongs in a local-AI runbook for a compliance-bound team? A runbook an end user can hand to auditors covers four things demos never do: rollback, which is how to return to the last known-good model, model provenance, which is what weights came from where and how you verified them, failure modes, which is what breaks and what users see, and operational boundaries, which is who can change what. It is the artifact that converts a pilot into a deployment. ## Understand why the auditor wants a runbook and not a demo A demo shows that the model works once, on one machine, with the person who built it sitting at the keyboard. An auditor asks a different question. They want evidence that the team can operate the model when that person is on vacation, and evidence that a bad change can be undone. The transcripts behind Theme 3 show the same pattern across regulated end users. The pilot gets approved on the demo, but the deployment gets approved on the runbook. The four sections map to the four questions an auditor asks in order. 1. Rollback answers "what do you do when a change goes wrong?" 2. Provenance answers "how do you know you are running what you think you are running?" 3. Failure modes answer "what does a user see when the system breaks?" 4. Operational boundaries answer "who is allowed to touch this, and how would you know if someone else did?" ## Write the rollback section first Rollback is the section auditors read first, because it is the one that limits damage. Write it as numbered steps that a teammate can execute without you. A rollback section for a llama.cpp deployment looks like this. 1. Model files live in `/srv/models/`. The file currently served is the one that `current.gguf` points to, and that pointer is a symlink. 2. The last known-good file is recorded in `/srv/models/KNOWN_GOOD`, which contains one filename and its SHA-256 hash. 3. To roll back, repoint the symlink to the known-good file and restart the server: ```bash ln -sfn /srv/models/bonsai-8b-q4_k_m-2026-05.gguf /srv/models/current.gguf systemctl restart llama-server ``` 4. Verify the rollback by checking the hash of the served file against `KNOWN_GOOD` and by running the smoke prompt in `/srv/models/smoke.txt`. The test for this section is blunt. A teammate should be able to execute it tonight without asking you anything. If a step says "ask Priya which file is safe", the section is not done. ## Record provenance so an auditor can verify it Provenance is a record of what weights you run, where they came from, and how you checked them. For each model version, record four fields. - The source, e.g., the [Bonsai 8B GGUF model card](https://huggingface.co/prism-ml/Bonsai-8B-gguf) and the exact file name and revision you downloaded. - The SHA-256 hash you computed after download, next to the hash the publisher lists. - The date you downloaded it and who did the download. - The license, copied from the model card, not paraphrased. Here is the worked arithmetic for the storage this policy costs you. A runbook that supports rollback needs at least three files on disk at once. Those are the current model, the last known-good model, and a candidate under test. An 8 billion parameter model at 4-bit quantization stores each weight in about half a byte, so the weights alone are 8 billion x 0.5 bytes = 4 GB. GGUF files at Q4_K_M keep some tensors at higher precision, so plan for roughly 5 GB per file. That is an estimate from the arithmetic, not a measured file size, so check the actual size on the model card. Three files at 5 GB is 15 GB, plus a few hundred MB for hashes, logs, and smoke-test outputs. Reserve 20 GB and the policy never forces you to delete the file you may need to roll back to. Do not write "verified" in a provenance record unless the record shows the command and both hashes. The transcripts show auditors treating an unverifiable claim as worse than no claim, because it suggests the rest of the runbook is also aspirational. ## List failure modes and operational boundaries The failure modes section is a table with three columns. The first column names what breaks, e.g., the model file is corrupt, the server runs out of memory, or the disk fills. The second column states what a user sees, e.g., an error page, a timeout, or a wrong answer with no error. The third column states the first response, which is usually the rollback procedure or a restart. The entries where the user sees a wrong answer with no error are the ones auditors care about most, so do not leave them out because they are uncomfortable. The operational boundaries section names people and permissions. State who can replace the model file, who can change the server configuration, and who can read the logs. Then state how a change outside those boundaries would be detected, e.g., a nightly job hashes `current.gguf` and alerts if the hash does not match the provenance record. A boundary with no detection is a request, not a boundary. ## Try it This takes about 25 minutes and runs on a laptop. No GPU is needed, because an 8B model at 4-bit quantization runs on CPU with about 6 GB of free RAM, just slowly. 1. Pick one local model you already run, even a laptop llama.cpp setup. If you have none, download one file from the [Bonsai 8B GGUF model card](https://huggingface.co/prism-ml/Bonsai-8B-gguf). 2. Create the skeleton: ```bash mkdir -p ~/runbook && cd ~/runbook printf "# Rollback\n\n# Provenance\n\n# Failure modes\n\n# Operational boundaries\n" > RUNBOOK.md ``` 3. Fill in provenance completely. Compute the hash and record it next to the publisher's hash: ```bash shasum -a 256 ~/models/bonsai-8b-q4_k_m.gguf ``` 4. Fill in rollback completely, with real paths and the exact restart command for your setup. Leave failure modes and boundaries as headings with one entry each. 5. Hand the rollback section to a teammate, or read it tomorrow yourself, and execute it without looking anything else up. ## Check yourself - **Could a teammate execute your rollback section tonight without asking you anything?** Expected answer: yes, because every step has a real path and a real command. If any step needs knowledge that is only in your head, the section is not done. - **How much disk should you reserve for an 8B model at Q4 with rollback support, and why?** Expected answer: about 20 GB, because you hold three files of roughly 5 GB each, which are current, known-good, and candidate, plus room for logs and hashes. - **What makes a provenance record verifiable rather than aspirational?** Expected answer: it shows the command that was run and both hashes, yours and the publisher's, so an auditor can rerun the check. - **Which failure mode entries do auditors weigh most?** Expected answer: the ones where the user sees a wrong answer with no error, because nothing alerts anyone. ## Next steps - Work through the [on-prem checklist runbook](https://yourwildcard.ai/docs/runbooks/on-prem-checklist.md) and merge its items into your skeleton. - Read the [regulated deployment case study](https://yourwildcard.ai/docs/case-studies/regulated-deployment.md) to see a runbook that passed an audit. - To understand the evaluation sequence your runbook feeds into, read [what a hospital CTO asks before running a model on-prem](/blog/hospital-cto-on-prem-ai-questions). When you can do this, you can author an auditor-ready runbook covering rollback, provenance, failure modes, and boundaries for a local deployment. --- --- title: "How do I make model cold starts faster?" description: "Cold starts split into four phases: GPU procurement, image load, weight load, and engine startup. Optimize each one separately to make scale-down safe." audience: inference-engineer pillar: foundational-concept book: inference-engineering chapter_ref: "Ch. 7, sec. 7.2.2 (pp. 188-190)" status: published last_reviewed: 2026-07-07 canonical_url: https://yourwildcard.ai/blog/make-model-cold-starts-faster --- # How do I make model cold starts faster? Cold start time splits into four phases you optimize separately: GPU procurement, image load, weight load, and engine startup. Smaller images, quantized weights (which also load faster), weight caches in the same datacenter, and cached compiled engines each attack one phase. Fast cold starts are what make scale-down safe. Slow ones force permanent over-provisioning. ## Understand why cold starts decide your bill A cold start is the time it takes to bring up a new replica of a model, from the moment the autoscaler asks for one to the moment it serves its first request. Philip Kiely's *Inference Engineering* (Ch. 7, sec. 7.2.2) ties this number directly to cost. He writes that "If you can't spin up replicas fast, it's hard to confidently scale down." The logic is simple. If a replica takes ten minutes to come up, you cannot shut replicas down when traffic dips, because a spike would leave users waiting ten minutes. So you keep extra replicas running at all times, and you pay for GPUs that sit idle. Cutting cold start time is what earns you the right to scale down. See the [autoscaling guide](https://yourwildcard.ai/docs/technical-guides/autoscaling.md) for how the scale-down delay setting depends on this number. ## Break the cold start into four phases Kiely lists four factors, and he is explicit that each one needs its own fix. - **GPU procurement.** The time your cloud provider takes to hand you a GPU node and attach it to your cluster. - **Image loading.** The time to pull the container image onto that node. - **Model loading.** The time to load the model weights into the container. - **Engine startup.** The time for the inference engine to start, including any compilation. Procurement is the phase you control least. Kiely notes it is mostly a function of your cloud provider, unless you keep a pool of warm nodes that you move between models. Node start time is also something you can negotiate in a contract. The other three phases are engineering work, and that is where the rest of this piece stays. ## Shrink what you have to move Image loading and weight loading are both the same problem. You are writing gigabytes, often hundreds of gigabytes, onto a fresh instance. Kiely gives two ways to speed that up: make the data smaller, or get more bandwidth. Smaller comes first because it is free. For the image, include only the steps and dependencies the server needs, and the image both builds and pulls faster. For the weights, quantization helps twice. A quantized model is cheaper to run, and the same smaller file also loads faster during a cold start. The [quantization guide](https://yourwildcard.ai/docs/technical-guides/quantization.md) covers how to quantize without losing quality. One older habit to drop. Teams used to bake the weights into the container image so there was one thing to cache. Kiely points out that modern models have tens or hundreds of billions of parameters, so the weights dwarf the image, and you should load them separately. ## Move the weights closer Bandwidth is the second lever, and where you load from sets the ceiling. If you pull weights from a third party such as Hugging Face, their egress speed limits you. An S3 bucket in another region adds network latency and data transfer costs. For models in the hundreds of billions of parameters, Kiely says you need gigabytes per second, and the way to get that is to load from a cache that sits physically in the same datacenter as the GPU instance. Here is the arithmetic for a small model, with the bandwidth numbers as assumptions rather than measurements. [Bonsai 8B](https://huggingface.co/prism-ml/Bonsai-8B-gguf) at FP16 is 2 bytes per parameter, so about 16 GB of weights. The Q4 GGUF is roughly 0.5 bytes per parameter plus overhead, call it 5 GB. Assume a 100 MB/s pull from a busy public source and a 1 GB/s pull from a datacenter local cache. ```text FP16 from public source: 16 GB / 0.1 GB/s = 160 s FP16 from local cache: 16 GB / 1.0 GB/s = 16 s Q4 from local cache: 5 GB / 1.0 GB/s = 5 s ``` The two fixes stack. Quantizing cut the file to about a third, and the local cache raised bandwidth ten times, so the phase went from an estimated 160 seconds to an estimated 5. Your numbers will differ, which is why the exercise below has you measure your own. ## Cache the compiled engine The last phase depends on which engine you run. Kiely notes that vLLM and SGLang start fast. TensorRT-LLM and optimized PyTorch models instead compile an engine targeted at the exact hardware, and those compilations often take several minutes. Paying that on every cold start would erase everything you saved in the other phases. The fix is to cache the built engine, and both TensorRT-LLM and PyTorch support this through their image caching mechanisms. The catch Kiely flags is strict. The cached engine only runs on an instance with exactly the same GPU type, CUDA version, and software dependencies as the environment it was built in. A cluster that mixes GPU types needs one cached engine per type. ## Try it This takes under 30 minutes. You need a machine that can run your model. A GPU box is ideal, but a Mac or CPU machine running llama.cpp or Ollama works for the image, weight, and engine phases. You cannot measure the procurement phase locally, so note it as zero and remember that in the cloud it is not. 1. Clear local state so the start is actually cold. For Docker, remove the image with `docker rmi ` and clear any weight cache volume. 2. Start a stopwatch and run the server, e.g., `docker run` for your serving image, or `ollama run bonsai-8b` for a local test. 3. Record three timestamps: when the image pull finishes, when the weight load finishes (most servers log this), and when `curl` to the completions endpoint first returns a valid response. 4. Write the total as four numbers: procurement (zero locally), image, weights, and engine. 5. Repeat the run with the [Bonsai 8B Q4 GGUF](https://huggingface.co/prism-ml/Bonsai-8B-gguf) instead of FP16 and compare the weight phase. You have done the exercise correctly when the four phase numbers add up to your total, and you can say which phase is largest. ## Check yourself 1. Name the four phases of a cold start. Expected answer: GPU procurement, image loading, model weight loading, and engine startup. 2. Your cold start is 8 minutes, and 6 of them are TensorRT-LLM compilation. What is the fix, and what is the constraint? Expected answer: cache the built engine. The cache only works on instances with the same GPU type, CUDA version, and dependencies as the build environment. 3. Why does quantization help cold starts and not just inference cost? Expected answer: the weight file is smaller, so it takes less time to write onto the instance. A Q4 file is roughly a quarter the size of FP16, so the weight phase shrinks by about the same ratio at the same bandwidth. 4. Why do slow cold starts cost money even when no replica is starting? Expected answer: the autoscaler cannot scale down safely if replacing a replica is slow, so you run extra replicas at all times and pay for idle GPUs. ## Next steps - [Autoscaling](https://yourwildcard.ai/docs/technical-guides/autoscaling.md) covers the scale-down delay and concurrency settings that decide when a cold start happens at all. - [Quantization](https://yourwildcard.ai/docs/technical-guides/quantization.md) explains how to shrink the weights, which speeds up the loading phase. - [Why does quantization speed up inference?](/blog/why-quantization-speeds-up-inference) covers the runtime side of the same file size reduction. When you can do this, you can profile a cold start into its four phases and select the highest-leverage optimization for your bottleneck. --- --- title: "What metrics should I monitor for an LLM inference service?" description: "Monitor seven families together: volume, sequence sizes, response codes, latency percentiles, replicas, utilization, and queue depth. Sizes explain spikes." audience: inference-engineer pillar: foundational-concept book: inference-engineering chapter_ref: "Ch. 7, sec. 7.4.3 (p. 203)" status: published last_reviewed: 2026-07-07 canonical_url: https://yourwildcard.ai/blog/metrics-to-monitor-llm-inference --- # What metrics should I monitor for an LLM inference service? Monitor seven families together: volume, request and response sizes, response codes, latency percentiles (TTFT, TPS, end-to-end), replica count, utilization, and queue depth. Sequence-length metrics are the inference-specific addition to standard SRE dashboards. Without them, "why is P99 up?" has no answer, because a traffic flood and a long-prompt flood look identical. ## Learn the seven families Kiely's Inference Engineering (Ch. 7, sec. 7.4.3) gives a list of what inference observability should measure. Here are the seven families in plain terms. - **Total volume.** How many requests the deployment receives. - **Request and response sizes.** The input and output sequence lengths, in tokens, for the requests you process. - **Response codes.** Counts of 2XX, 4XX, and 5XX codes from the model server. - **Latency percentiles.** Time to first token, tokens per second, and end-to-end latency, each reported at P50, P90, and P99 rather than as an average. - **Replica count.** How many instances are serving traffic, and how many are still starting up. - **Utilization.** How busy the CPU, host memory, GPU, and GPU memory are. - **Queue depth.** For services that queue work, how many requests are waiting to be processed. Five of these appear on any web service dashboard. The one that is specific to inference is request and response sizes. A normal web request costs about the same to serve no matter what the user typed. An inference request does not. The prefill work grows with the input length, and the decode work grows with the output length, so a request with a 50,000 token prompt costs far more than one with a 500 token prompt even though both count as one request. ## Read the metrics together, not one at a time The book's point about this list is that the metrics are interdependent. Kiely writes that a latency spike "could come from request volume, but it could also come from long input sequences", and that seeing the metrics together lets an engineer understand not only what is happening but why. This is why a single latency alert is not enough. When P99 TTFT climbs, at least three different causes produce the same graph. - More requests arrived, so batches filled up and requests waited. - The same number of requests arrived, but the prompts got longer, so each prefill took more compute. - A replica died, so the remaining replicas absorbed its traffic. The latency chart looks identical in all three cases. You only tell them apart by cross-referencing volume, input sequence length, and replica count. Each cause also has a different fix. More traffic calls for scaling out, which the [autoscaling guide](https://yourwildcard.ai/docs/technical-guides/autoscaling.md) covers. Longer prompts call for a look at what upstream code changed, e.g., a retrieval step that started stuffing more documents into the context. A dead replica calls for a restart and a hardware check. Response codes and queue depth complete the picture. A rise in 5XX codes alongside a latency spike points at crashing or overloaded replicas. A growing queue with flat utilization points at a stuck consumer rather than a capacity problem. ## Work a staged incident with the numbers Here is a made-up incident with example dashboard values, so the arithmetic is the point rather than the specific numbers. Your alert fires because P99 end-to-end latency went from 3 seconds to 14 seconds at 10:00. You open the dashboard and read the other families for the same window. - Volume: 120 requests per minute before 10:00, 118 after. Flat. - Replica count: 4 before, 4 after. Flat. - P99 input length: 1,200 tokens before 10:00, 24,000 tokens after. - Response codes: still almost all 2XX. - GPU utilization: up from 55 percent to 95 percent. Volume and replicas are flat, so this is not a traffic flood and not lost capacity. The input length percentile is the metric that moved. A 24,000 token prompt has 20 times the prefill work of a 1,200 token prompt, because prefill work scales with the input length. That extra compute per request is what pushed utilization to 95 percent and stretched the latency, even though the request count never changed. The next question goes to whoever owns the client code, e.g., did a retrieval pipeline start attaching whole documents instead of snippets at 10:00. Now flip one number. If input length had stayed flat and volume had jumped from 120 to 600 requests per minute, the same latency graph would mean a traffic flood, and the fix would be more replicas, not a client-side investigation. One latency chart, two opposite responses, and only the sequence-length metric tells you which one you are in. One more note from the same section. Kiely advises against building inference monitoring as its own silo. Send these metrics into the observability and alerting tools your team already uses, e.g., Grafana or Datadog, so inference data sits next to the rest of the application. ## Try it This takes under 30 minutes and does not need a GPU. [Bonsai 8B](https://huggingface.co/prism-ml/Bonsai-8B-gguf), PrismML's open-weight 1-bit model, runs on a laptop CPU or Apple Silicon through llama.cpp. Start a server with the [llama.cpp recipe](https://yourwildcard.ai/docs/build-and-run/bonsai-llamacpp.md). ```bash llama-server -m Bonsai-8B-Q1_0.gguf --port 8080 ``` Save this as `monitor.py` and run it with `python3 monitor.py`. It sends 50 requests, logs TTFT, output length, and status code for each one, and prints a P50 and P99 summary. ```python import json, time, requests, statistics URL = "http://127.0.0.1:8080/v1/completions" prompts = [f"Explain topic {i} in one paragraph." for i in range(50)] rows = [] for p in prompts: payload = {"prompt": p, "max_tokens": 150, "stream": True} start = time.perf_counter() ttft, tokens, status = None, 0, None with requests.post(URL, json=payload, stream=True) as r: status = r.status_code for line in r.iter_lines(): if not line.startswith(b"data: ") or line == b"data: [DONE]": continue chunk = json.loads(line[len(b"data: "):]) if chunk["choices"][0].get("text"): if ttft is None: ttft = time.perf_counter() - start tokens += 1 rows.append({"ttft": ttft, "tokens": tokens, "status": status}) ttfts = sorted(r["ttft"] for r in rows if r["ttft"] is not None) lens = sorted(r["tokens"] for r in rows) codes = {} for r in rows: codes[r["status"]] = codes.get(r["status"], 0) + 1 def pct(xs, q): return xs[min(len(xs) - 1, int(q * len(xs)))] print(f"requests: {len(rows)}") print(f"status codes: {codes}") print(f"TTFT P50/P99: {pct(ttfts, 0.50)*1000:.0f} ms / {pct(ttfts, 0.99)*1000:.0f} ms") print(f"out len P50/P99: {pct(lens, 0.50)} / {pct(lens, 0.99)} tokens") ``` Then stage a long-prompt flood. Change a few of the prompts to include several thousand tokens of pasted text and run the script again. Watch P99 TTFT move while the request count stays at 50. That is the exact signature from the incident above. ## Check yourself 1. P99 latency spikes. Which two metrics do you cross-reference first to distinguish more traffic from longer inputs? Expected answer: total request volume and input sequence length. If volume rose and lengths are flat, it is traffic. If volume is flat and lengths rose, it is longer prompts. 2. Which of the seven families would not appear on an ordinary web service dashboard, and why does inference need it? Expected answer: request and response sizes. The cost of serving an inference request grows with its token counts, so two requests are not interchangeable units of work the way they are for a typical web endpoint. 3. Queue depth is growing but GPU utilization is low. What does that combination suggest? Expected answer: requests are not reaching the replicas, e.g., a stuck consumer or a routing problem, rather than a shortage of compute. ## Next steps - [Observability](https://yourwildcard.ai/docs/technical-guides/observability.md) shows how to export these metrics from a production deployment. - [Autoscaling](https://yourwildcard.ai/docs/technical-guides/autoscaling.md) covers the response when the diagnosis is genuinely more traffic, and how replica count and queue depth drive scaling decisions. - [Why report P99 latency, not the average](/blog/why-report-p99-latency-not-average) explains the percentile choice this post takes for granted. When you can do this, you can instrument an inference service with the seven metric families and diagnose a staged latency incident from the dashboard alone. --- --- title: "Why is model selection the biggest inference optimization?" description: "Smaller models are always faster and cheaper, so the smallest model that passes your evals beats any runtime trick. Eval design comes before serving." audience: researcher pillar: foundational-concept book: inference-engineering chapter_ref: "Ch. 1, secs. 1.3-1.3.2 (pp. 31-33)" status: published last_reviewed: 2026-07-07 canonical_url: https://yourwildcard.ai/blog/model-selection-biggest-inference-optimization --- # Why is model selection the biggest inference optimization? All else equal, smaller models are always faster and cheaper, so choosing the smallest model that passes your evals beats any runtime trick. A model swap can outdo months of kernel work, which makes eval design upstream of every serving decision and the highest-leverage skill in the stack. ## Understand why size wins before any runtime trick Every inference optimization works within limits set by the model you picked. Kiely's Inference Engineering (Ch. 1.3) argues that when hardware, runtime, and optimizations are held equal, a model with fewer parameters is always faster and cheaper to run than a model with more parameters. This is not an empirical finding that could go the other way on your workload. It follows from arithmetic. A decoder generates one token at a time, and each token requires reading every weight from memory. An 8B model quantized to 4 bits is about 5 GB of weights. A 70B model at the same precision is about 40 GB. On a machine with 100 GB/s of memory bandwidth, the ceiling is roughly 20 tokens per second for the 8B model and roughly 2.5 tokens per second for the 70B model. These are upper bounds, not benchmarks, but the ratio holds because it comes from the sizes themselves. Now compare that ratio to what runtime work buys. A better attention kernel or a smarter batching scheduler typically improves throughput by tens of percent. The model swap above is an 8x change. That is why Kiely puts model choice ahead of the runtime engine and the speculation algorithm. Months of kernel engineering cannot recover the gap between a 70B model and an 8B model that both do the job. ## Make evals the gate, not the afterthought The catch in "the smallest model that works" is the word "works". You cannot know which models work without measuring them on your task. Kiely (Ch. 1.3.1) defines evals as the practice of systematically measuring model intelligence, and separates them from public benchmarks like MMLU, which measure common tasks rather than your product. Public benchmarks are useful for shortlisting candidates, but Kiely notes they have become saturated and even gamed, and cites Goodhart's Law as the reason. On measuring your own task, he writes that "there is no substitute for directly measuring how a model performs for your application". This ordering is the whole point of this post. If evals decide which model passes, and the model decides your latency and cost floor, then eval design sits upstream of every serving decision. A researcher who can build a sharp 20-prompt eval set has more control over production cost than one who can shave 10 percent off a kernel. Kiely (Ch. 1.3.2) gives text-to-SQL as the extreme case. General coding models with hundreds of billions of parameters write good SQL, but the task is constrained enough that a fine-tuned model of a few billion parameters can match them on that one task. He is explicit that most domains will not support a reduction that large. ## Work the numbers on a model swap Here is the arithmetic for a made-up but realistic workload, with every input stated so you can rerun it with your own numbers. Suppose your service handles 1 million requests per day, each producing 400 output tokens. That is 400 million output tokens per day. If a 70B model and an 8B model both pass your eval, the 8B model reads one eighth the weights per token, so on the same hardware it needs roughly one eighth the GPU time for decode. A fleet that cost you 8 GPUs now costs you 1, or the same 8 GPUs now serve 8 times the traffic at lower latency per request. The same logic applies if you buy tokens instead of GPUs. Per-token prices scale with model size because the provider faces the same arithmetic. The exact ratio varies by provider, so treat the 8x above as an estimate for this example rather than a market fact. The decision procedure is short. 1. Write the eval set and the pass threshold before you test anything. 2. Score the candidate models against it. 3. Pick the smallest model that passes. 4. Only then spend time on quantization, batching, and kernels. Kiely adds one constraint at step 3. Stick with popular model architectures, because inference engines vary in how well they support each architecture, and an exotic model can lock you out of the optimizations you wanted the small model for. ## Try it Build a 10-prompt eval set for one task you care about, then score a small model and a larger one in llama.cpp. Budget 30 minutes. First, write 10 prompts for your task into a file, one prompt per line, and write down your pass threshold before running anything. For example, "passes" might mean 8 of 10 outputs are correct by your own judgment. ```bash # Install llama.cpp (macOS) brew install llama.cpp # Download one small and one larger instruct model in GGUF format, # e.g. a 3B and a 14B from Hugging Face, then run each prompt: while IFS= read -r p; do llama-cli -m small-3b-q4_k_m.gguf -p "$p" -n 256 --temp 0 -no-cnv done < prompts.txt while IFS= read -r p; do llama-cli -m larger-14b-q4_k_m.gguf -p "$p" -n 256 --temp 0 -no-cnv done < prompts.txt # Measure speed on your hardware: llama-bench -m small-3b-q4_k_m.gguf -m larger-14b-q4_k_m.gguf ``` Score both models against your threshold and record the tokens per second from llama-bench. Then write one paragraph deciding whether the smaller model passes, citing your scores and your threshold. ## Check yourself 1. Did your pass threshold exist in writing before you ran either model? If you wrote it after seeing outputs, the eval is measuring your reaction to the models rather than the task. 2. Does your written decision cite the threshold and the scores, e.g., "the 3B model scored 9 of 10 against a threshold of 8, so it passes"? A decision that says the small model "felt fine" fails this check. 3. Can you state the speed and cost difference you observed, e.g., the tokens per second from llama-bench for each model? 4. If the small model failed, can you say which prompts it failed on? Those prompts tell you whether fine-tuning could close the gap, per Kiely's text-to-SQL example. ## Next steps - [Benchmarking guide](https://yourwildcard.ai/docs/technical-guides/benchmarking.md) covers how to measure latency and throughput properly once you have picked a model. - [llama.cpp in the ecosystem](https://yourwildcard.ai/docs/ecosystem/llama-cpp.md) explains where llama.cpp fits and how to get models for it. When you can do this, you can construct a task-specific eval set and use it to justify selecting a smaller model, citing accuracy, latency, and cost. --- --- title: "Why are MoE models fast locally but not on servers?" description: "MoE models activate a fraction of parameters per token, so batch-1 local inference is cheap. Batched server traffic activates nearly all experts." audience: inference-engineer pillar: foundational-concept book: inference-engineering chapter_ref: "Ch. 2, sec. 2.2.4 (pp. 53-54)" status: published last_reviewed: 2026-07-07 canonical_url: https://yourwildcard.ai/blog/moe-fast-locally-slow-on-servers --- # Why are MoE models fast locally but not on servers? MoE models activate only a fraction of their parameters per token, e.g., 22B of 235B, which makes single-request local inference cheap. But batched server traffic activates nearly all experts across the batch, and that erases the advantage unless you add expert parallelism. The same architecture helps on a laptop and hurts on a shared server. ## Separate resident memory from per-token compute A mixture of experts (MoE) model replaces the one large feed-forward matrix in each transformer block with many smaller matrices called experts. A small router picks a few experts for each token at each layer, and only those experts do work for that token. This creates two different numbers, and the whole question turns on keeping them apart. - Resident memory is the total parameter count. Every expert must sit in RAM or VRAM, because the router may pick any of them for the next token. This number never shrinks. - Per-token compute is the active parameter count. Only the chosen experts are read and multiplied for a given token. Philip Kiely's *Inference Engineering* (Ch. 2, sec. 2.2.4) uses Qwen3-235B-A22B as the example. The model has 235 billion total parameters, and 22 billion are activated per request. The name encodes both numbers. ## See why batch 1 favors MoE When you run a model locally, you decode one request at a time. Decode at batch 1 is limited by memory bandwidth, not by arithmetic, because each step reads the weights once to produce one token. For a dense model, each token reads all the weights. For an MoE model, each token reads only the attention weights, the router, and the chosen experts. Fewer bytes read per token means more tokens per second on the same hardware. This is why Kiely writes that MoE models are "highly efficient for single-request local inference". The cost is that you still pay full resident memory. A machine that runs the model at all must hold every expert, even though each token touches few of them. ## See why a batched server loses the advantage A production server does not decode one request at a time. It batches many requests and decodes one token for each of them in the same step. Each token picks its own experts, and different tokens pick different ones. Kiely's warning is that in batched inference, different requests activate different experts, so you "should expect almost all of the model parameters to be active" unless you shard the experts across GPUs with expert parallelism (his section 5.4.2). Once the batch touches nearly every expert per step, the server reads nearly all the weights per step anyway, so the bandwidth saving that made the model fast locally is gone. What remains is a model that costs 235B parameters of memory while a dense competitor with similar quality might cost far less. Expert parallelism is the fix. It places whole experts on different GPUs, so each GPU reads only its own experts and the batch's work spreads across the machines. The [parallelism guide](https://yourwildcard.ai/docs/technical-guides/parallelism.md) covers how to turn it on. ## Work one example These figures are derived from the model's published architecture, not measured benchmarks. - Qwen3-235B-A22B has 235B total parameters and 22B active per token. In FP8, at one byte per parameter, the resident weights are about 235 GB no matter the batch size. - At batch 1, each decode step reads about 22 GB of weights instead of 235 GB. That is roughly 10.7 times fewer weight bytes per token than a dense 235B model, so decode is roughly that much faster on the same memory bandwidth. - Kiely notes the router picks 8 of 128 experts at each of the model's 94 layers. Assume for a moment that routing is uniform, which real routing is not, so treat this as an estimate. For one token, a given expert at a given layer sits idle with probability 120/128, which is about 0.94. - With a batch of 32 tokens per step, that expert sits idle with probability 0.94 to the power of 32, which is about 0.13. So about 87 percent of the experts at each layer do work in every step. - At batch 64 the idle probability falls to about 0.02, so about 98 percent of experts are active per step. At batch 1 the server reads 22 GB per step. At batch 64 it reads close to the full 235 GB per step. The per-token saving has almost vanished, and the memory bill never went away. ## Try it This takes about 20 minutes with [Ollama](https://ollama.com) installed. You are reproducing the batch-1 case, where MoE wins, and measuring what it costs in memory. Pick an MoE model and a dense model with a similar active parameter count. Qwen3-30B-A3B has about 3B active parameters, so a dense 3B model is a fair partner. ```bash ollama run qwen3:30b-a3b --verbose "Explain the KV cache in three sentences." ollama run llama3.2:3b --verbose "Explain the KV cache in three sentences." ``` The `--verbose` flag prints the eval rate in tokens per second. While a model is loaded, run `ollama ps` in another terminal to see its memory footprint. Expect the MoE model to need roughly 20 GB of RAM or VRAM at 4-bit, while the dense 3B model needs about 2 GB. Those sizes are estimates from parameter count and quantization, so record your own numbers. If your machine cannot fit the 30B MoE, [Bonsai 8B](https://huggingface.co/prism-ml/Bonsai-8B-gguf) is a dense model that fits in about 5 GB and still shows the dense side of the comparison. The [hardware requirements guide](https://yourwildcard.ai/docs/getting-started/hardware-requirements.md) maps model sizes to machines. Then write one paragraph explaining the gap you measured. The check is that your paragraph keeps resident memory, meaning all experts loaded, separate from per-token compute, meaning only the active experts read. If the two numbers blur together in your explanation, rewrite it. ## Check yourself 1. An MoE model has 235B total and 22B active parameters. How much memory does it need at batch 1 in FP8, and how many weight bytes does one decode step read? Expected answer: about 235 GB resident, because every expert must be loaded, but each step reads only about 22 GB, which is why batch 1 is fast. 2. Why does batching erase the MoE speed advantage? Expected answer: each token in the batch picks its own experts, so a large batch activates nearly all experts per step, and the server reads close to the full weights per step just like a dense model. 3. What is the infrastructure prerequisite for serving a large MoE model efficiently? Expected answer: expert parallelism, which shards whole experts across GPUs so each GPU reads only its local experts and the batch's expert traffic spreads out. 4. Your teammate proposes an MoE architecture for a 7B model that serves one user on a laptop. Is MoE the right call? Expected answer: probably not, because Kiely notes models under 32B, and especially under 8B, tend to run efficiently as dense models, and the extra total parameters raise the memory bill. ## Next steps - [Parallelism](https://yourwildcard.ai/docs/technical-guides/parallelism.md) covers expert parallelism flags for the major inference engines. - [Hardware requirements](https://yourwildcard.ai/docs/getting-started/hardware-requirements.md) maps resident model size to the machines that can hold it. - [Tensor vs pipeline vs expert parallelism](/blog/tensor-vs-pipeline-vs-expert-parallelism) covers when expert parallelism beats the other strategies. When you can do this, you can predict how an MoE model's advantage changes between batch-1 and batched serving and state the infrastructure prerequisite for server-side MoE. --- --- title: "Should I use one omni-modal model or a pipeline of small specialists?" description: "Small specialists often beat the same capability inside a VLM at a fraction of the size. The trade is that a pipeline buys accuracy and cost with orchestration." audience: ml-product-team pillar: end-user-case-study book: inference-engineering chapter_ref: "Ch. 6, sec. 6.1.2 (p. 159)" status: published last_reviewed: 2026-07-07 canonical_url: https://yourwildcard.ai/blog/ocr-model-vs-vlm-for-receipts --- # Should I use one omni-modal model or a pipeline of small specialists? Omni models are convenient, but small specialists often beat the same capability baked into a VLM at a fraction of the size. A dedicated OCR model that fits in a few hundred megabytes can outscore a 7B VLM on receipt extraction while using a tenth of the RAM. The trade is that each pipeline stage scales on its own, so you buy accuracy and cost with orchestration work. ## Know what each path is An omni-modal model is one model that accepts several kinds of input and produces several kinds of output. A vision language model, or VLM, is the common case for receipts. Kiely's Inference Engineering (sec. 6.1) explains that a VLM is two modules: a standard LLM plus a small vision encoder that turns raw images into image tokens. You send it a receipt photo and a prompt, and it writes the extracted fields as text. A pipeline of specialists splits that job into stages. An OCR model reads the pixels and outputs plain text. Then a small text model, or even a parser, turns that text into structured fields. Kiely (sec. 6.1.2) notes that production VLM systems already look like this in practice, with individual preprocessors for extracting data from PDFs, reading text from images via OCR, and transcribing audio. ## Weigh accuracy against convenience The book's verdict on omni models is measured. Their blend of modalities gives unique capabilities, but Kiely (sec. 6.1.2) states that "smaller specialized models are often faster and more accurate within specific domains". He gives text recognition as the example. Many VLMs have OCR ability trained into their image processing, but those abilities "lag behind dedicated optical character recognition (OCR) models" that are generally a fraction of the size (Inference Engineering, sec. 6.1.2). Receipts are exactly that specific domain. The task is not open-ended visual reasoning. It is reading printed text and mapping it to a fixed schema. A specialist trained only on text recognition spends all of its capacity on that one skill, while a 7B VLM spends most of its capacity on general language and vision ability you do not use. ## Do the memory and token arithmetic Here is one worked comparison. The parameter counts are exact arithmetic, and the runtime figures are estimates for planning, not benchmarks. The VLM path. A 7B model in FP16 needs 7 billion parameters times 2 bytes, which is 14 GB of weights. A 4-bit quantized file needs about 7 times 0.5, which is roughly 3.5 GB, plus KV cache and runtime overhead. Kiely's rule of thumb (sec. 6.1) is that one high-resolution image adds about a thousand visual tokens to the input sequence. So each receipt costs a prefill of roughly 1,000 image tokens plus your prompt before the model writes a single output token. The pipeline path. Tesseract, a widely used open OCR engine, installs in tens of megabytes and typically runs in well under 1 GB of RAM (estimate). Its output for a receipt is a few hundred characters of plain text. If you then hand that text to a small LLM such as [Bonsai 8B](https://huggingface.co/prism-ml/Bonsai-8B-gguf) to structure it, the LLM reads roughly 300 text tokens instead of 1,000 image tokens, and you only need the LLM stage at all when a parser cannot handle the format. Note what the arithmetic implies for scale. Kiely (sec. 6.1.2) says each pipeline component must be individually optimized for speed and should scale independently to avoid bottlenecks. If OCR is fast and structuring is slow, you add replicas only to the structuring stage. With one omni model, you can only scale the whole thing. The [cost modeling guide](https://yourwildcard.ai/docs/technical-guides/cost-modeling.md) shows how to price both shapes. The pipeline's cost is orchestration. Two stages means two deployments, two sets of metrics, a queue between them, and two places where a bad input can fail. At low traffic, that overhead can cost more engineer time than the VLM's extra RAM costs in hardware. The bake-off below gives you the numbers to find your own crossover point. ## Try it Run both paths on five receipt screenshots in about 25 minutes. No GPU is needed. On a laptop, the VLM will be slow, and that is part of the lesson. First, collect five receipt screenshots as PNG files and write out the ground-truth text for the total line of each one by hand. Time the specialist with Tesseract. ```bash brew install tesseract time tesseract receipt1.png out1 cat out1.txt ``` Time a local VLM with llama.cpp. This downloads a 7B vision model, which needs roughly 6 GB of memory for the 4-bit file plus the vision encoder. ```bash brew install llama.cpp time llama-mtmd-cli -hf ggml-org/Qwen2.5-VL-7B-Instruct-GGUF \ --image receipt1.png -p "Transcribe every line of this receipt exactly." ``` Repeat both commands for all five receipts. Score each path with exact-match on the total line: the extracted text either matches your ground truth exactly or it does not. Record three numbers per path: exact matches out of five, median seconds per receipt from `time`, and peak memory from Activity Monitor while the command runs. If Tesseract misreads a receipt, that is a finding too. Crumpled or photographed receipts favor stronger OCR models or the VLM, and clean digital screenshots favor the specialist. Where the specialist wins on accuracy and speed, the remaining question is whether your traffic justifies running two services, and the [inference stack overview](https://yourwildcard.ai/docs/concepts/inference-stack.md) shows where each stage sits. ## Check yourself 1. Why does a dedicated OCR model often beat a VLM at reading receipts? Expected answer: Kiely (sec. 6.1.2) reports that VLM text recognition lags behind dedicated OCR models that are a fraction of the size, because the specialist spends all of its capacity on one domain. 2. Roughly how much memory do the 4-bit weights of a 7B model need, and how did you get the number? Expected answer: about 3.5 GB, from 7 billion parameters times half a byte, before KV cache and overhead. 3. Why does a receipt image cost more input tokens than the OCR text of the same receipt? Expected answer: one high-resolution image adds about a thousand visual tokens (sec. 6.1), while the OCR text of a receipt is a few hundred text tokens. 4. Your OCR stage runs at 20x the speed of your structuring stage. What does independent scaling let you do? Expected answer: add replicas only to the structuring stage, instead of scaling one omni model that bundles both jobs. ## Next steps - [Cost modeling](https://yourwildcard.ai/docs/technical-guides/cost-modeling.md) turns your bake-off numbers into a monthly cost for each architecture. - [The inference stack](https://yourwildcard.ai/docs/concepts/inference-stack.md) explains where OCR, VLM, and LLM stages sit in a serving system. - [Why do images blow up my context window?](/blog/images-blow-up-context-window) goes deeper on the image token arithmetic used above. When you can do this, you can run a specialist-versus-generalist bake-off and justify a pipeline architecture from measurements. --- --- title: "Ollama vs llama.cpp — which should I use for local inference?" description: "llama.cpp is the runtime and Ollama is tooling wrapped around it. Ship a feature with Ollama's defaults, or tune flags, quants, and benchmarks with llama.cpp." audience: inference-engineer pillar: ecosystem-player book: transcript-theme chapter_ref: "Theme 2: end users vs ecosystem members; Kiely Ch. 0 tooling layer" status: published last_reviewed: 2026-07-07 canonical_url: https://yourwildcard.ai/blog/ollama-vs-llama-cpp --- # Ollama vs llama.cpp — which should I use for local inference? Ollama and llama.cpp sit on different layers of the same stack, so this is not a fair fight between two rivals. llama.cpp is the runtime that loads a GGUF file and produces tokens. Ollama is tooling wrapped around that runtime, and it adds model management, an API server, and defaults that work without tuning. Ship a feature with Ollama. Tune flags, quantization formats, or benchmarks with llama.cpp directly. Most of the debate comes from layer confusion, not from a capability gap. ## Place each tool on the stack An inference stack has three layers. The runtime runs one model on one machine as fast as the hardware allows. The infrastructure scales that model across machines. The tooling is the interface an engineer uses to work with the other two. Kiely's *Inference Engineering* (Ch. 0) argues that the tooling layer exists to give an engineer enough control to run inference confidently and enough abstraction to work productively. On that map, llama.cpp is a runtime. It compiles to a binary, loads a quantized model file, and exposes every knob the engine has, e.g., how many layers to place on the GPU. Ollama is tooling. For GGUF models it runs a llama.cpp engine under the hood. On top of that engine it adds: - A model registry, so `ollama pull llama3.1:8b` fetches weights, the chat template, and default parameters as one named package. - A local API server that speaks a stable HTTP interface, so your application code does not care which engine sits below. - Lifecycle management, e.g., Ollama loads a model on first request and unloads it after five minutes of idle time by default. So the question "Ollama or llama.cpp" is really "do I want the runtime bare, or with tooling around it". ## Decide who you are on this project The same split shows up in how people relate to an ecosystem. An end user consumes a tool to ship something else. An ecosystem member works on the tool layer itself, reads its source, files issues, and tunes its internals. - If you are an end user of local inference, Ollama is the tooling built for you. You trade control for convenience, and on most days that trade is correct because the defaults are fine and your job is the feature. - If you are acting as an ecosystem member, you want llama.cpp directly. You need to pin the exact build, choose the quantization type of the KV cache, and hold every flag constant while you measure one change. The switch point in a project is concrete. You start with Ollama to prove the feature works at all. You move to a bare `llama-server` the first time you need a setting that Ollama does not expose, or the first time you need a benchmark number you can defend. ## Work through the cost of the convenience Here is a worked example with derived numbers, not benchmarks. Suppose your model file is 4.7 GB on disk, which is a common size for an 8B model at 4-bit quantization. Suppose your SSD reads at 3 GB/s. Treat both numbers as assumptions for the arithmetic. Loading the file into memory takes 4.7 / 3, which is about 1.6 seconds, plus engine startup on top. With a bare `llama-server`, you pay that once when you start the process, and the model then stays resident until you stop it. Your first request after startup is fast. With Ollama's default lifecycle, the model unloads after five minutes idle. If your app gets one request every ten minutes, every request pays the load again, so every user waits the extra 1.6 seconds and more before the first token. That is not Ollama being slow. That is a tooling default chosen for a laptop that runs many models, applied to a service that runs one. Ollama lets you change it with the `keep_alive` setting, and knowing that this knob exists is exactly the layer awareness this post is about. When two tools wrap the same runtime, a speed difference between them is usually a settings difference, e.g., context length, GPU layer count, or model residency. Before you blame either tool, print the effective settings from both and compare them. ## Try it Run the same GGUF model through Ollama and through a bare `llama-server`, then compare what you can control. This takes under 30 minutes with a model you already have. 1. Pull and time the Ollama path. The first `run` includes model load. ```bash ollama pull llama3.1:8b time ollama run llama3.1:8b "Reply with one word." ollama show llama3.1:8b --parameters ``` 2. Find the GGUF blob Ollama downloaded, or use any GGUF file you have, and serve it bare. On macOS, `brew install llama.cpp` provides `llama-server`. ```bash llama-server -m /path/to/model.gguf -c 4096 -ngl 99 --port 8080 time curl -s http://localhost:8080/v1/chat/completions \ -H "Content-Type: application/json" \ -d '{"messages":[{"role":"user","content":"Reply with one word."}],"max_tokens":5}' ``` 3. Read `llama-server --help` next to the Ollama parameter list. Write down three settings you can control in one tool but not the other, e.g., the KV cache quantization type per server flag, a sampler that only one side exposes, or the idle unload time. 4. Note the time to first token for both paths, once cold and once warm, and note which differences come from settings rather than from the engine. ## Check yourself 1. Which layer of the three-layer stack does each tool occupy? Expected answer: llama.cpp is a runtime and Ollama is tooling that wraps a runtime, and neither is infrastructure. 2. At what point in a project would you switch from Ollama to a bare `llama-server`? Expected answer: when you need a setting Ollama does not expose, or when you need benchmark numbers with every flag pinned. 3. Your service on Ollama shows a slow first token every ten minutes. What do you check first? Expected answer: the `keep_alive` idle unload, because the model is reloading from disk, and the runtime itself is not the problem. ## Next steps - Read the ecosystem pages on [Ollama](https://yourwildcard.ai/docs/ecosystem/ollama.md) and [llama.cpp](https://yourwildcard.ai/docs/ecosystem/llama-cpp.md) for the settings each one exposes. - Read the concept page on the [inference stack](https://yourwildcard.ai/docs/concepts/inference-stack.md) for the full three-layer map. - Read [What are the three layers of an inference stack?](/blog/three-layers-of-an-inference-stack) to practice placing failures, not just tools, on the same map. When you can do this, you can place a local inference tool on the runtime and tooling axis and choose one based on how much control you need versus how much convenience you want. --- --- title: "Should I run AI on-device or in the cloud? The eight-factor tradeoff" description: "Local inference wins on latency, privacy, offline independence, and zero marginal cost. It loses on device capability, thermals, fragmentation, and battery." audience: ml-product-team pillar: end-user-case-study book: inference-engineering chapter_ref: "Ch. 3, sec. 3.5 Local Inference" status: published last_reviewed: 2026-07-07 canonical_url: https://yourwildcard.ai/blog/on-device-vs-cloud-ai-inference --- # Should I run AI on-device or in the cloud? The eight-factor tradeoff Local inference wins on four factors: latency, privacy, offline independence, and zero marginal cost. It loses on four others: device capability, thermal limits, hardware fragmentation, and battery drain. Those eight factors are a ready-made scorecard. Run a feature such as meeting transcription through all eight and one factor is usually decisive, which turns a philosophical debate into a checklist. ## Start with the eight factors from the book Philip Kiely's *Inference Engineering* (Ch. 3, sec. 3.5) defines local inference as "running AI model inference directly on the end user's device" instead of on a server. He then lists four advantages and four weaknesses, and together they cover almost every argument a product team will have about this decision. The four advantages of running on the device are these. - **Latency.** There is no network round trip, which saves tens or even hundreds of milliseconds per request. - **Independence.** The feature keeps working with no internet connection, and server outages and traffic spikes cannot break it. - **Privacy.** The user's data never leaves their device. - **Cost.** Datacenter GPUs are expensive, and inference on the user's own hardware costs the developer nothing per request. The four weaknesses are these. - **Device capability.** Kiely notes that even a high-end desktop offers a fraction of the speed of a datacenter GPU, and phones struggle with models above one or two billion parameters. - **Thermal limits.** A phone or laptop cools itself far worse than a datacenter rack, so sustained inference forces the device to slow down. - **Fragmentation.** Your users run many combinations of chips, operating systems, and driver versions, and you must support all of them. - **Battery.** Inference is a heavy workload and drains a phone or laptop battery quickly. Most debates about on-device AI go in circles because each person argues from one factor. The fix is to stop debating and score all eight for the specific feature in front of you. ## Score a real feature: meeting transcription Meeting transcription is a good test case because Kiely calls out transcription as one of the workloads that fits on a phone. Speech models are latency-sensitive, and some are small enough to run in real time on a modern device. Say your product transcribes meetings for 1,000 business users, and each user records about 5 hours of meetings a month. Here is the scorecard. | Factor | Local or cloud? | Why | | --- | --- | --- | | Latency | Local | Live captions need words on screen as they are spoken, and a network round trip adds delay to every chunk. | | Independence | Local | Meetings happen on planes and in conference rooms with bad wifi. | | Privacy | Local | Meeting audio contains salaries, legal issues, and unreleased plans. | | Cost | Local | The arithmetic below shows the cloud bill grows with every user. | | Device capability | Cloud | A large transcription model is more accurate, but small ones fit on phones. | | Thermal limits | Cloud | A one hour meeting is sustained load, so the device heats up. | | Fragmentation | Cloud | You must test across iOS, Android, and several chip generations. | | Battery | Cloud | An hour of on-device transcription costs meaningful battery. | Now the cost row in numbers. A cloud transcription API priced at 0.6 cents per audio minute, which is a typical published rate as of this writing, gives you this bill: ``` 1,000 users x 5 hours x 60 minutes = 300,000 minutes per month 300,000 minutes x $0.006 = $1,800 per month, or $21,600 per year ``` The on-device version has zero marginal cost. And the capability weakness is mild here, because a small transcription model such as Whisper small has about 244 million parameters, which is well under the one to two billion parameter ceiling Kiely gives for phones. ## Circle the decisive factor The scorecard reads 4 to 4, but the factors do not weigh the same. For meeting transcription, privacy is decisive. A business customer whose legal team forbids sending meeting audio to a third party will not buy the cloud version at any price or any latency. The other seven factors adjust how good each option is. Privacy decides whether one of the options is allowed to exist. Here is the test that proves it. Imagine privacy off the table, because your customers have all approved cloud processing. Now the decision flips. The cloud version transcribes with a larger and more accurate model, drains no battery, heats no phones, and needs no per-device testing, and $1,800 a month is a small bill for 1,000 paying users. The decision flipped when one factor left, so that factor was driving it and the other seven were not. Kiely closes the section by saying the future is not local or cloud but both together. Small models and quick queries run on the device while heavy workloads stay in the cloud. A transcription feature can run a small model live on the device and send audio for a higher quality second pass only when the user opts in. One more warning from the book. When you score device capability, score your median user's hardware, not yours. Kiely points out that an AI enthusiast has the latest phone, while a median user has an older device with weaker components. If your scorecard only works on this year's flagship phone, the local option loses the capability row for most of your users. ## Try it This takes about 20 minutes. 1. Pick one AI feature in your product, or one you are planning. 2. Draw a table with the eight factors as rows: latency, independence, privacy, cost, device capability, thermal limits, fragmentation, and battery. 3. For each row, write "local" or "cloud" and one sentence of evidence. For the cost row, do the arithmetic the way the example above does, with your own user count and usage. 4. Circle the single factor that would decide the question on its own. 5. Run the removal test. Imagine the circled factor is gone and ask whether your decision flips. If it does not flip, you circled the wrong factor. Find the one that makes it flip. If your feature involves speech, you can also measure the capability row directly instead of guessing. Install whisper.cpp and time a small model on the oldest laptop in your office: ```bash git clone https://github.com/ggml-org/whisper.cpp cd whisper.cpp && make ./models/download-ggml-model.sh small ./main -m models/ggml-small.bin -f samples/jfk.wav ``` If the transcription runs faster than the audio plays, the capability row supports local for that device. ## Check yourself 1. Your feature is a photo tagger for a consumer app, and you circled cost as decisive. Cloud tagging would cost you $50 a month. Did you circle the right factor? Expected answer: no. Removing a $50 bill does not flip any decision, so cost was not driving it. Rescore and find the factor that flips it. 2. A teammate says on-device always wins because it is free. Which rows answer them? Expected answer: device capability, thermal limits, fragmentation, and battery. Free per request is one row out of eight, and the four weakness rows are where free gets paid for. 3. Your scorecard came out 4 to 4. What does that tell you? Expected answer: nothing by itself. The count does not decide, the decisive factor does. Apply the removal test to each candidate until you find the one that flips the decision. ## Next steps - Check whether your target machines can run a local model at all in [hardware requirements](https://yourwildcard.ai/docs/getting-started/hardware-requirements.md). - Set up local transcription with [whisper.cpp](https://yourwildcard.ai/docs/ecosystem/whisper-cpp.md). - If your decisive factor was device capability, read [how much VRAM do I need to run an LLM?](/blog/how-much-vram-to-run-an-llm) to size the model your hardware can hold. When you can do this, you can apply the eight-factor scorecard to a real feature and identify the decisive constraint. --- --- title: "What's the difference between online and offline inference?" description: "Online inference optimizes for latency because a user is waiting. Offline batch inference optimizes for throughput because fewer GPUs means cheaper." audience: inference-engineer pillar: foundational-concept book: inference-engineering chapter_ref: "Ch. 1, sec. 1.2.2 (pp. 29-30)" status: published last_reviewed: 2026-07-07 canonical_url: https://yourwildcard.ai/blog/online-vs-offline-inference --- # What's the difference between online and offline inference? Online inference optimizes for latency because a user is waiting. Offline batch inference optimizes for throughput because fewer GPUs means cheaper. The trade is direct, so the same model often deserves two deployments. Real-time dictation and back-catalog transcription can share Whisper weights but nothing else about their configuration. ## Separate online from offline before you configure anything An online workload has a person waiting on the other end of the request. Chat, code completion, and voice agents are all online. If the response is slow, the person notices and the product feels bad. So an online deployment should be configured for low latency, e.g., small batches and a short queue. An offline workload has no one waiting. The job runs over a fixed pile of inputs and the only person who cares about timing is whoever pays the GPU bill. Kiely's Inference Engineering (Ch. 1, sec. 1.2.2) lists examples like transcribing a back catalog of podcasts, embedding a document set on a schedule, and cleaning a corpus for training. For these jobs, each individual request can be slow. What you want is the highest number of requests finished per hour of GPU time, because that number sets the cost. Kiely calls latency versus throughput one of the primary tradeoffs in the field. In his words, lower latency "makes your application faster, but higher throughput makes it cheaper at scale" (Inference Engineering, sec. 1.2.2). The mechanism is batching. A GPU finishes more total work per hour when it processes many requests together, but each request in a big batch waits on the others, so each one finishes later. You can read why in our [batching guide](https://yourwildcard.ai/docs/technical-guides/batching.md). ## Run the same model twice when both workloads exist The model is not the deployment. Kiely uses Whisper, a speech-to-text model, as the example. A dictation app needs words on screen while the speaker is still talking, so it feeds the model small audio chunks and accepts that the GPU sits partly idle between them. A back-catalog job has the whole archive on disk already, so it packs the GPU with as much audio as fits and lets every request wait its turn. The book's advice is that when both use cases have enough volume, you should run two separate deployments of the same weights, one configured for latency and one for throughput. A single deployment has to pick one side of the trade. If it picks latency, the batch job burns GPU hours on idle capacity. If it picks throughput, the dictation user watches a spinner. ## Work the cost arithmetic on one example The speeds below are made up to show the arithmetic. Measure your own before deciding anything. Suppose you have 10,000 hours of archived audio to transcribe, a GPU costs $2 per hour to rent, and you measure these two speeds on your hardware: - The latency-tuned deployment, with small chunks and a batch size near 1, transcribes 5 hours of audio per GPU-hour. - The throughput-tuned deployment, with large batches, transcribes 40 hours of audio per GPU-hour. The batch job costs are then: - Latency-tuned config: 10,000 / 5 = 2,000 GPU-hours, which is $4,000. - Throughput-tuned config: 10,000 / 40 = 250 GPU-hours, which is $500. Running the archive through the online deployment overpays by 8 times, and the ratio holds at any GPU price because both numbers scale with it. The mistake in the other direction does not show up as money. It shows up as latency. A throughput-tuned deployment waits to fill a batch before it starts, so the dictation user waits seconds for the first word instead of a fraction of a second. One deployment cannot win both metrics, which is why the book recommends two. ## Try it You can feel both modes in about 20 minutes with [whisper.cpp](https://yourwildcard.ai/docs/ecosystem/whisper-cpp.md) and one podcast episode. No GPU is needed, because the base model runs at usable speed on a laptop CPU, and whisper.cpp uses the GPU on Apple silicon automatically. Build whisper.cpp and fetch a model: ```bash git clone https://github.com/ggml-org/whisper.cpp && cd whisper.cpp cmake -B build && cmake --build build -j ./models/download-ggml-model.sh base.en ``` Convert your episode to the format whisper.cpp reads: ```bash ffmpeg -i episode.mp3 -ar 16000 -ac 1 episode.wav ``` First, the online-style run. Split the audio into 10 second chunks to imitate audio arriving live, then time how long the first chunk takes. That time is your latency to the first words. ```bash ffmpeg -i episode.wav -f segment -segment_time 10 -c copy chunk_%03d.wav time ./build/bin/whisper-cli -m models/ggml-base.en.bin -f chunk_000.wav for f in chunk_*.wav; do ./build/bin/whisper-cli -m models/ggml-base.en.bin -f "$f"; done ``` Time the whole loop too. Then the offline-style run, the entire file as one job: ```bash time ./build/bin/whisper-cli -m models/ggml-base.en.bin -f episode.wav ``` Record three numbers: latency to first words from the first chunk, wall-clock per audio-hour for the chunked loop, and wall-clock per audio-hour for the single batch run. The chunked run gets you words within seconds of "audio arriving", and the single run finishes the whole episode in less total time, because each chunk pays model startup and padding overhead that the batch run pays once. ## Check yourself 1. Which configuration wins on latency to first word, and why? Expected answer: the chunked run, because it starts transcribing after 10 seconds of audio exist instead of waiting for the whole file. 2. Which configuration wins on wall-clock per audio-hour, and why? Expected answer: the single batch run, because the per-chunk overhead is paid once and the hardware stays busy. 3. Your company runs only the latency-tuned deployment and pushes the archive job through it. Using the worked example above, what does that cost? Expected answer: 8 times the GPU spend of a throughput-tuned deployment, and the multiple is whatever ratio your own two measured speeds give. 4. When does one deployment stop being enough? Expected answer: when both an online and an offline use case have enough volume, per Kiely (sec. 1.2.2), because one configuration must sacrifice either the user's latency or the batch job's cost. ## Next steps - [whisper.cpp](https://yourwildcard.ai/docs/ecosystem/whisper-cpp.md) covers the tool used in the exercise. - [Batching](https://yourwildcard.ai/docs/technical-guides/batching.md) explains the mechanism that trades latency for throughput. - [Batch size and the latency-throughput tradeoff](/blog/batch-size-latency-throughput-tradeoff) works the same trade at the level of a single server's batch size. When you can do this, you can configure the same model for latency and for throughput and quantify the cost of using the wrong mode. --- --- title: "Why should I pin exact dependency versions in my inference container?" description: "Inference dependency chains are long and fragile. Pinning exact versions preserves a known good build, and one loose pin can break tomorrow's rebuild." audience: inference-engineer pillar: ecosystem-player book: inference-engineering chapter_ref: "Ch. 7, sec. 7.1.1 (pp. 181-182)" status: published last_reviewed: 2026-07-07 canonical_url: https://yourwildcard.ai/blog/pin-dependency-versions-inference-containers --- # Why should I pin exact dependency versions in my inference container? Inference dependency chains, from the CUDA toolkit to torch to the engine build, are long and fragile, and the ecosystem breaks often. Pinning exact versions in your container preserves a known good build, and a single `>=` pin can turn tomorrow's rebuild into an incident. Day zero model support often runs on nightly builds that you must re-pin once stable releases land. ## Know what your image actually contains Philip Kiely's *Inference Engineering* (Ch. 7, sec. 7.1.1) opens with the core claim: "Dependency chains for inference are long and fragile." Getting to a working build is hard, so the point of the container is to preserve a known good build in an ecosystem where breaking changes are common. Kiely lists four kinds of runtime components inside an inference image. - The CUDA toolkit. The specific versions of CUDA, cuDNN, and drivers that are compatible with the rest of your stack. - Python packages, e.g., `torch`, `transformers`, and `diffusers`. - The inference engine, e.g., a specific version of vLLM, SGLang, or TensorRT-LLM. - System packages, e.g., `ffmpeg` when you work with audio or video models. Every one of these has its own release schedule and its own idea of which versions of the others it supports. An unpinned dependency means the version is chosen at build time, not by you. Your image works today because one specific combination of all four kinds happens to be compatible. A rebuild that picks even one newer version is a different combination that nobody has tested. ## Pin exact versions, not ranges Kiely's Figure 7.2 shows the rule in three lines. `transformers` is marked "No". `transformers>=4.40.0` is also marked "No". Only `transformers==4.56.2` is marked "Yes". A range pin like `>=` reads as cautious, but it still lets the resolver pick a version that did not exist when you tested the build. In practice a `>=` pin behaves like no pin at all on the day a breaking release ships. The payoff Kiely describes is repeatability. Once a fully pinned image builds successfully, every later build resolves to the same versions and produces the same runtime behavior. Tools like uv, poetry, and pip check the pinned set for conflicts and fail the build with an error instead of silently picking something new. A failed build at build time is a much cheaper problem than a broken model server at request time. The book also tells you where to start. vLLM and SGLang publish official base images for active releases, and Kiely recommends starting from one of those proven images rather than building your own from scratch. See [/docs/ecosystem/vllm](https://yourwildcard.ai/docs/ecosystem/vllm.md) for the vLLM images. ## Count how much a short requirements file leaves open Here is the arithmetic that surprises most people the first time. Suppose your requirements file has 5 lines: `torch`, `transformers`, `vllm`, `fastapi`, and `uvicorn`, none of them pinned. Build the image and run `pip freeze` inside it, and you will see the full resolved set. On a typical vLLM image that set runs well past 100 packages. Call it 120 for this example, which is an estimate, not a benchmark. Now subtract. You wrote 5 lines, so 120 minus 5 leaves 115 packages that you never named. Those are transitive dependencies, meaning packages that your packages depend on. Pinning your 5 lines to exact versions still leaves those 115 free to drift, because your direct pins do not fix what `torch` or `vllm` pulls in next week. That is 115 out of 120 packages, or about 96 percent of the image's Python dependencies, chosen fresh on every rebuild. The fix is to pin the resolved set, not just your direct lines. Run `pip freeze` in the working container and use that full output as your requirements file, or use a lock file from uv or poetry, which records every transitive version. After that, a rebuild resolves all 120 packages to the versions you already tested. ## Re-pin nightlies once stable releases land Kiely points out that breaking changes are especially common around newly released models. When a new model like DeepSeek is announced, the whole ecosystem races to offer day zero support, and inference engineers often build images from overnight builds or other developer pre-releases because no stable release supports the model yet. Those early builds are more prone to bugs, and Kiely notes they often need to be rebuilt on stable releases in the days and weeks after the model drop. So the workflow has two steps, not one. First, pin the exact nightly build you tested, e.g., a dated dev version, so your day zero image is at least reproducible. Second, put a dated task in your tracker to swap that pin for the stable release once it ships, rebuild, and re-run your evaluation. A nightly pin without a follow-up task is a known good build today and unowned risk in a month. ## Try it This takes under 30 minutes and needs no GPU, because you are auditing files, not serving a model. If you cannot run Docker locally, you can still do the audit on the requirements file alone. Pick one Dockerfile or requirements file from a service you own. Then run these steps. ```bash # 1. List every unpinned line in your requirements file grep -vE '==' requirements.txt # 2. Check the base image tag in your Dockerfile grep '^FROM' Dockerfile # 3. In the running container, capture what is actually installed docker exec pip freeze > pinned.txt ``` Step 1 shows every line that is not pinned to an exact version. Step 2 catches a `latest` or missing tag on the base image, which is the same drift problem one layer down. Step 3 gives you the exact versions that are working right now. Replace your requirements file with `pinned.txt`, or pin each unpinned line to the version shown there, and rebuild to confirm the image still builds and starts. Write down two numbers: how many lines were unpinned, and how many packages `pip freeze` reported. The gap between them is your transitive exposure. ## Check yourself 1. Your requirements file says `vllm>=0.8.0` and the image built fine last month. Why can today's rebuild fail? Expected answer: `>=` lets the resolver pick a newer vllm release than the one you tested, and a newer release can change or break behavior. Only an exact `==` pin reproduces the tested build. 2. Every direct dependency in your file is pinned with `==`. What can still drift on a rebuild? Expected answer: transitive dependencies, the packages your packages pull in, unless you pin the full resolved set with `pip freeze` output or a lock file. 3. You built a day zero image for a brand new model on a nightly engine build. What two things should you do? Expected answer: pin the exact nightly version so the image is reproducible, and schedule a re-pin to the stable release once it ships, with a rebuild and re-evaluation. 4. Name the four kinds of runtime components Kiely says an inference image includes. Expected answer: the CUDA toolkit versions, Python packages, the inference engine version, and system packages such as ffmpeg. ## Next steps - Read the vLLM ecosystem page at [/docs/ecosystem/vllm](https://yourwildcard.ai/docs/ecosystem/vllm.md), including the official base images that Kiely recommends starting from. - Build a first pinned image around a real endpoint with [/docs/getting-started/first-inference](https://yourwildcard.ai/docs/getting-started/first-inference.md), e.g., serving [Bonsai 8B](https://huggingface.co/prism-ml/Bonsai-8B-gguf). - See how engine choice interacts with your container in [vLLM vs SGLang vs TensorRT-LLM](/blog/vllm-vs-sglang-vs-tensorrt-llm), since the engine version is one of the four components you are pinning. When you can do this, you can convert a drifting inference build into a fully pinned, reproducible one and explain the nightly to stable re-pin workflow. --- --- title: "Why is LLM prefill compute-bound but decode memory-bound?" description: "Prefill loads weights once for big matrix-matrix math. Decode reloads all weights per token for small vector-matrix math, so decode is memory bound." audience: inference-engineer pillar: foundational-concept book: inference-engineering chapter_ref: "Ch. 2, sec. 2.4.2 (pp. 63-66)" status: published last_reviewed: 2026-07-07 canonical_url: https://yourwildcard.ai/blog/prefill-compute-bound-decode-memory-bound --- # Why is LLM prefill compute-bound but decode memory-bound? Prefill loads the model weights once and multiplies large matrices, so it performs many operations for every byte it reads from memory. Decode reloads all the weights for each new token to do one small vector times matrix multiply, so it performs few operations per byte. In the book's worked example, a decode attention step reaches about 62 operations per byte, and an H100 can support 295. ## Compare arithmetic intensity to the GPU's ops:byte ratio A GPU has two main resources. Compute is how many floating point operations it can do per second. Memory bandwidth is how many bytes it can move per second. Divide the first by the second and you get the GPU's ops:byte ratio. Kiely's Inference Engineering (Ch. 2, sec. 2.4.1) works this out for an H100 in FP16. The chip does 989 teraFLOPS of dense compute against 3.35 TB/s of memory bandwidth, which gives a ratio of about 295. Arithmetic intensity is the same ratio measured for one algorithm instead of for the hardware. You count the total operations the algorithm performs and divide by the total bytes it reads and writes. Then you compare the two numbers. - If the algorithm's intensity is above the GPU's ratio, the GPU runs out of compute first. The algorithm is compute bound. - If the intensity is below the ratio, the GPU runs out of bandwidth first. The algorithm is memory bound. Prefill and decode land on opposite sides of this line. In prefill, the model reads the weights once and multiplies whole matrices of input tokens against them, so each byte loaded feeds many operations. In decode, the model reads every weight again for each single token, and each token only needs one vector times matrix multiply, so each byte loaded feeds few operations. This is why prefill sets your time to first token and decode sets your tokens per second. ## Work through the decode attention example Kiely's Inference Engineering (Ch. 2, sec. 2.4.2, pp. 63-66) makes this concrete with one decode attention step. The setup is a 128-dimensional attention head (d = 128) over a sequence of 4096 tokens (N = 4096), using the plain attention algorithm with no optimizations and FP16 values at two bytes each. The matrices have these sizes. Q, K, and V are each N by d. The score matrix S and the softmax output P are each N by N. The output O is N by d. The algorithm runs in three steps, and each step loads data from memory, computes, and writes the result back. Summing the reads and writes gives the total memory traffic, and summing the multiply and add work gives the total compute. - Total memory traffic is 8N² + 8Nd bytes. With these numbers that is about 138 million bytes. - Total compute is 4N²d + 3N² operations. With these numbers that is about 8.6 billion operations. Divide the two and you get about 62 operations per byte. The H100 can support 295. The book puts it plainly: this example "illustrates the general principle that decode is memory bound." The GPU spends most of the step waiting on memory, and its compute units sit idle. Kiely notes that "calculating arithmetic intensity like this is an academic exercise, not a routine task" for inference engineers, but doing it once builds intuition you will use every time you size hardware. The prefill version of the same math looks different because prefill processes all input tokens in one pass. The weight reads stay roughly the same, but the compute multiplies by the full sequence length, so the intensity rises far above the hardware ratio and the step becomes compute bound. ## Use the split to pick the right optimization Once you know which side of the line a phase sits on, you know which fixes can help it and which cannot. - **Batching.** Batching helps decode because many requests share one read of the weights, so the compute per byte goes up. Kiely notes that batching makes decode less memory bound for exactly this reason. - **Quantization.** Quantization shrinks each weight, so decode moves fewer bytes per token. It helps decode more than prefill because decode's cost is bytes moved, not operations done. - **Disaggregation.** Some systems run prefill and decode on separate machines, because the two phases want different hardware. A compute-heavy phase and a bandwidth-heavy phase do not share one box well. If you optimize compute for a memory-bound step, nothing improves. The book states this directly for bottlenecks in general, and decode is the most common place people learn it the hard way. ## Try it Redo the book's calculation for your own model. You need the head dimension d from your model's config file and two context lengths. This runs in any Python shell in under a minute. ```python def intensity(N, d): mem = 8 * N**2 + 8 * N * d # bytes moved, FP16 ops = 4 * N**2 * d + 3 * N**2 # floating point operations return ops / mem for N in (4096, 32768): print(N, round(intensity(N, 128), 1)) ``` Swap in your own d and your own two context lengths. Then look up your GPU's dense FP16 FLOPS and memory bandwidth, divide them to get its ops:byte ratio, and state which side each result lands on. On an H100 the ratio is about 295, so both of the values above land on the memory-bound side. ## Check yourself 1. Why does quantization help decode more than prefill? Expected answer: decode's cost is the bytes of weights it moves per token, and quantization cuts those bytes. Prefill is limited by operations, and quantization does not remove operations. 2. A decode attention step has an intensity of 62 and your GPU's ops:byte ratio is 295. Which resource is idle? Expected answer: compute is idle, because the step hits the bandwidth limit long before the compute limit. 3. What does batching change in the intensity calculation? Expected answer: it raises the operations performed per byte of weights read, because one weight read now serves many requests. ## Next steps - Read the [prefill and decode concept page](https://yourwildcard.ai/docs/concepts/prefill-decode.md) for how the two phases fit into a serving stack. - Read the [quantization guide](https://yourwildcard.ai/docs/technical-guides/quantization.md) to act on the bytes-moved insight from this post. When you can do this, you can compute arithmetic intensity for an attention step and predict whether it is compute bound or memory bound on given hardware. --- --- title: "How should I order my prompt to maximize prefix cache hits?" description: "The cache match runs from the first token to the first non-repeated token, so variable content early kills all savings after it. Push novel tokens late." audience: ml-product-team pillar: foundational-concept book: inference-engineering chapter_ref: "Ch. 5, sec. 5.3.1 (pp. 137-138)" status: published last_reviewed: 2026-07-07 canonical_url: https://yourwildcard.ai/blog/prompt-ordering-for-prefix-cache-hits --- # How should I order my prompt to maximize prefix cache hits? The cache match runs from the first token until the first non-repeated token. So variable content early in the prompt, e.g., a user name, a date, or the query, destroys all savings after that point. Push novel tokens as late as possible. Reordering one prompt template can cut cost and latency with no model changes. ## Learn the one rule that decides your hit rate Prefix caching lets the server reuse the stored keys and values for tokens it has already processed in an earlier request. The match starts at the very first token of the prompt and ends at the first token that differs from the earlier request. Nothing after that point is reused, even if the rest of the prompt is identical. Kiely's Inference Engineering (Ch. 5, sec. 5.3.1) shows this with two four-token prompts. "SF weather today ?" and "NYC weather today ?" share their last three tokens, but the first token differs, so the cache reuses nothing. The book's Figure 5.8 caption states the point plainly: the first tokens are different, so it does not matter that the next three are the same. The reason is that the model is autoregressive. The stored value for each token depends on every token before it. One new token at position 5 makes the stored values for positions 6 and beyond wrong for the new request, so the server must recompute them. From this, Kiely gives one rule for prompt layout: "ensure that novel tokens are as late in your context as possible." Everything else in this post follows from that rule. ## Sort your template from most stable to most variable Take your prompt template and sort its parts by how often they change across requests. - Put the system prompt, tool definitions, and few-shot examples first. These are identical on every request. - Put shared documents or retrieved context next, since many requests in a session reuse them. - Put per-request fields last: the date, the user's name, session IDs, and the user's question. The common mistake is a header like "Today is 2026-07-07. You are a support agent for..." at the top of the template. The date changes daily and a timestamp changes every request, so the match ends within the first few tokens and the entire system prompt behind it is recomputed at full price. The fix is a one-line move. Keep the instructions at the top and append the dynamic line at the end, e.g., "For reference, today is 2026-07-07." The model still sees the date, and the whole fixed block in front of it now matches the cache. ## Count the tokens your current ordering throws away Here is a worked example with a common template shape. Suppose your template has a 12-token header that contains the current date, then 1,800 tokens of fixed instructions and examples, then about 90 tokens of user input. The numbers are an example, not a benchmark. - With the date first, the match ends inside the header. At best a few tokens hit the cache, so roughly 1,890 of about 1,900 input tokens are recomputed on every request. - With the date and user input moved to the end, the 1,800 fixed tokens match on every request after the first. Only about 100 tokens are recomputed. The reorder recovers about 1,790 cached tokens per request. At 10,000 requests per day, that is about 17.9 million input tokens per day that the server no longer prefills. If your API charges $3.00 per million tokens on a miss and $0.30 on a hit, those tokens drop from $53.70 per day to $5.37 per day, and those are example prices. On your own server the same reorder shows up as lower time to first token, because prefill on the fixed block is skipped. Watch the chat template, not just your own text. The serving framework wraps your messages in role tags and may inject the date or other fields into the rendered prompt. Print the rendered token sequence once and check where the first differing token lands. ## Try it You can A/B the two orderings in under 30 minutes with llama.cpp on a laptop. No GPU is needed. An 8B model at 4-bit needs about 6 GB of memory, and prefill will be slower on CPU than on a GPU or Apple silicon, which makes the cache effect easier to see. Start a server with [Bonsai 8B](https://huggingface.co/prism-ml/Bonsai-8B-gguf). llama-server reuses the matching prompt prefix by default: ```bash brew install llama.cpp llama-server -hf prism-ml/Bonsai-8B-gguf --port 8080 ``` Then compare time to first token for the two orderings. Each ordering is sent twice, and the second send is the one to compare: ```python import time, requests fixed = "You are a support agent. " + ("Policy line to follow. " * 300) def ttft(system): start = time.time() with requests.post("http://localhost:8080/v1/chat/completions", json={"stream": True, "messages": [ {"role": "system", "content": system}, {"role": "user", "content": "How do I reset my password?"}]}, stream=True) as r: for line in r.iter_lines(): if line and line != b"data: [DONE]": return time.time() - start for name, tpl in [("date first", lambda d: f"Today is {d}. " + fixed), ("date last", lambda d: fixed + f" Today is {d}.")]: ttft(tpl("2026-07-07")) # warm the cache print(name, "repeat TTFT:", ttft(tpl("2026-07-08"))) # date changed ``` With the date first, the changed date breaks the match at the front, so the repeat request runs prefill on the whole fixed block. With the date last, the fixed block matches and only the tail is recomputed, so the repeat time to first token should drop by a large factor. Then apply the same reorder to one of your production templates and rerun the comparison. ## Check yourself 1. Two prompts are identical except for one token in the middle. How much of the second prompt can hit the cache? Expected answer: only the tokens before the differing token, because the match runs from the first token to the first non-repeated token and nothing after it is reused. 2. Your template is a 15-token greeting with the user's name, then 2,000 fixed tokens, then the question. About how many cached tokens does this ordering throw away per repeat request? Expected answer: about 2,000, since the name breaks the match inside the first 15 tokens and the fixed block behind it is recomputed. 3. Why does moving the user's question from the top to the bottom of the prompt change cost but not what the model sees? Expected answer: the model still receives every token, so the content is unchanged, but the fixed block now forms an unbroken matching prefix and the server skips prefill on it. ## Next steps - Read the [prefix caching guide](https://yourwildcard.ai/docs/technical-guides/prefix-caching.md) for how the cache is stored and evicted on a real server. - Read the [cost modeling guide](https://yourwildcard.ai/docs/technical-guides/cost-modeling.md) to turn your measured hit rate into a per-request cost estimate. - For the pricing side of the same mechanism, read [Why do API providers charge less for cached input tokens?](/blog/why-cached-input-tokens-cost-less) from this series. When you can do this, you can restructure a prompt template for maximal prefix reuse and measure the resulting TTFT and cost delta. --- --- title: 'Qwen3.6 27B versus Ternary Bonsai 27B' description: "Compare the base Qwen3.6 27B model with PrismML's ternary representation across architecture, size, quality, runtime, and use cases." audience: local-ai-builder pillar: bonsai-27b status: published last_reviewed: 2026-07-15 canonical_url: https://yourwildcard.ai/blog/qwen36-27b-vs-ternary-bonsai-27b --- # Qwen3.6 27B versus Ternary Bonsai 27B Qwen3.6 27B is the base multimodal model. Ternary Bonsai 27B keeps that architecture but changes the language weight representation and runtime path. Qwen's full precision language weights need about 54 GB, while PrismML reports a 5.9 GB ideal ternary representation and about 7.2 GB for its current deployed GGUF language model. ## What stays the same - The language model has 64 blocks and a 262,144 token context. - The architecture mixes linear attention and full attention. - The model accepts text and images and returns text. - The Qwen3.6 training and post training behavior is the starting point. ## What changes PrismML maps the language weights into ternary values and applies group scales inside low bit kernels. The vision tower remains a separate 4 bit component. The deployment uses PrismML compatible MLX, llama.cpp, or CUDA paths instead of a normal FP16 runtime. The smaller representation reduces memory traffic but also changes benchmark results. PrismML reports 80.49 for ternary against 85.07 for its FP16 Qwen reference. ## Questions people ask ### Can I use the normal Qwen Transformers command with the ternary model? Use the runtime and format described by PrismML. The low bit files need kernels that understand their packing. ### Does Ternary Bonsai change the context limit? PrismML documents the same 262,144 token maximum, though available memory can set a lower practical limit on a local machine. ## Sources - [Qwen3.6 27B base model card](https://huggingface.co/Qwen/Qwen3.6-27B) - [PrismML Bonsai 27B documentation](https://docs.prismml.com/models/bonsai-27b) - [PrismML Bonsai 27B whitepaper](https://github.com/PrismML-Eng/Bonsai-demo/blob/main/bonsai-27b-whitepaper.pdf) - [PrismML formats and runtime support](https://docs.prismml.com/download/formats) ## Related Bonsai 27B lessons - [What is Ternary Bonsai 27B?](/blog/what-is-ternary-bonsai-27b) - [How good is Ternary Bonsai 27B?](/blog/ternary-bonsai-27b-quality-benchmarks) - [Why Ternary Bonsai 27B is called a 1.58 bit model](/blog/ternary-bonsai-27b-bits-per-weight) The benchmark numbers on this page describe one checkpoint, runtime, machine, and test shape. Reproduce the test on the hardware and workload you plan to use before making a product decision. --- --- title: 'How to run Ternary Bonsai 27B with MLX' description: 'Install MLX LM, download Ternary Bonsai 27B, generate text, and avoid the most common Apple silicon setup errors.' audience: local-ai-builder pillar: bonsai-27b status: published last_reviewed: 2026-07-15 canonical_url: https://yourwildcard.ai/blog/run-ternary-bonsai-27b-mlx --- # How to run Ternary Bonsai 27B with MLX Install the current mlx-lm package, then pass the Hugging Face model ID to mlx_lm.generate or mlx_lm.server. We loaded and generated from the model with stock mlx-lm 0.31.3 on an M4 Pro Mac. The first run downloads the 8.49 GB MLX bundle. ## Install and generate Run the command on an Apple silicon Mac. The files are cached after the first download, so later starts should reuse them. PrismML's written MLX guide still mentions its fork, while the current Hugging Face model page gives the stock install shown above. The stock path worked in our July 14, 2026 test with mlx-lm 0.31.3. ```bash uv tool install mlx-lm mlx_lm.generate \ --model prism-ml/Ternary-Bonsai-27B-mlx-2bit \ --prompt "Explain KV cache growth in one paragraph." ``` ## Check the result - Confirm the model ID is exactly `prism-ml/Ternary-Bonsai-27B-mlx-2bit`. - Watch free disk space during the first download. - Record package versions when you benchmark because MLX performance can change between releases. - Use a short prompt first, then increase context after the model is stable. ## Questions people ask ### Do I need PrismML's MLX fork? Not on our tested setup. Stock mlx-lm 0.31.3 loaded the model and returned a completion on an M4 Pro Mac. Record your version because PrismML's written guide and current Hugging Face instructions do not agree. ### Why does the first request take so long? The first run may still be downloading files or loading 8.49 GB of model data into memory. Wait for the download and model load to finish before measuring response speed. ## Sources - [PrismML MLX guide](https://docs.prismml.com/run/mlx) - [PrismML Bonsai 27B documentation](https://docs.prismml.com/models/bonsai-27b) - [Ternary Bonsai 27B MLX model card](https://huggingface.co/prism-ml/Ternary-Bonsai-27B-mlx-2bit) ## Related Bonsai 27B lessons - [What Mac do you need for Ternary Bonsai 27B?](/blog/ternary-bonsai-27b-mac-requirements) - [How to serve Ternary Bonsai 27B through an OpenAI compatible API](/blog/ternary-bonsai-27b-openai-api) - [Ternary Bonsai 27B performance on an M4 Pro Mac](/blog/ternary-bonsai-27b-performance-m4-pro) The benchmark numbers on this page describe one checkpoint, runtime, machine, and test shape. Reproduce the test on the hardware and workload you plan to use before making a product decision. --- --- title: "Should you run the demo before you finish the paper?" description: "For systems papers, yes. Run the demo first, then read the paper to debug your gaps. The claims land as answers to problems you just hit, not abstractions." audience: researcher pillar: end-user-case-study book: transcript-theme chapter_ref: "Theme 6: Demo-driven learning" status: published last_reviewed: 2026-07-07 canonical_url: https://yourwildcard.ai/blog/run-the-demo-before-finishing-the-paper --- # Should you run the demo before you finish the paper? For a systems paper, yes. Get the demo or a small implementation running first, then read the paper to debug your gaps. Reading after you apply sticks because the paper's claims land as answers to problems you just hit, not as abstractions to memorize. The transcripts behind Theme 6 describe teams that work this way, and they report the paper reads differently, and faster, after the artifact runs. ## Treat the paper as a debugger, not a textbook A systems paper describes a running artifact. The authors built something, measured it, and then wrote the prose. When you read the prose first, you receive the claims in the reverse order from how they were produced, and you have no way to tell which sentences are load-bearing. Every claim looks equally plausible, so you file all of them at the same shallow depth and forget most of them within a week. The transcripts behind Theme 6 show the same pattern across teams. An engineer reads a paper about an inference technique, nods along, and cannot answer a basic question about it a month later. The same engineer runs the technique on a local model for twenty minutes, gets a confusing number in the logs, goes back to the paper to explain the number, and can still explain the mechanism months later. The paper did not change. The question the engineer brought to it did. This is why the order is run first, read second. Running the demo generates specific confusions, e.g., a metric in the output you cannot interpret. The paper then answers those confusions one by one, and each answer attaches to something you saw with your own eyes. ## Stop reading at the point of first confusion The loop has a concrete shape. 1. Read only the abstract and skim the figures, no more than ten minutes. You want to know what the system claims, not how it works. 2. Get the smallest runnable version of the technique working. Prefer a demo you can run on a laptop over the authors' full setup. 3. Write down every output you cannot explain. These are your questions. 4. Now read the paper in full, hunting for the sentence that explains each question. The failure mode this prevents is finishing the paper with zero questions. Zero questions does not mean full understanding. It usually means the reading produced no contact with the system at all, so nothing pushed back on your assumptions. ## Work through one misreading and its correction Here is the kind of correction the loop produces, using speculative decoding as the technique. The [speculative decoding guide](https://yourwildcard.ai/docs/technical-guides/speculative-decoding.md) covers the mechanism in full. On a first pass through a speculation paper, many readers file the headline speedup as a fixed property of the method, e.g., "this technique makes decoding about 2 to 3 times faster." Then they run a demo and see the speedup swing with the prompt, and the paper suddenly reads differently. The speedup was never a constant. It is arithmetic over the acceptance rate, which the running system prints and the first reading skipped. The arithmetic is short. Suppose the draft model proposes 4 tokens per step and the target model accepts each proposed token with probability 0.7. Treat 0.7 as an assumption for the example, not a measurement. The expected number of tokens produced per target model forward pass is (1 - 0.7^5) / (1 - 0.7) = (1 - 0.168) / 0.3 = about 2.8 tokens. Without speculation, one forward pass produces exactly 1 token. So 2.8 tokens per pass is an upper bound of about a 2.8x speedup, before you subtract the cost of running the draft model. Drop the acceptance rate to 0.4 and the same formula gives about 1.6 tokens per pass, and the draft cost can eat most of that. The claim you misread as a benchmark was a function, and the running demo is what forces you to see its input. Do not quote a speedup from your demo as if it generalizes. It is one measurement on your hardware, your prompt, and your model pair. Report it that way or not at all. ## Try it This takes about 25 minutes of active work, plus a model download of a few gigabytes. No GPU is needed. The commands run on CPU and run faster on Apple Silicon or a CUDA GPU. The technique is prefix caching, because its smallest demo is two commands. 1. Skim the prefix caching section of a paper or guide you have only read, for ten minutes at most. Write one sentence stating what you think it claims. 2. Download a small open model, e.g., [Bonsai 8B in GGUF format](https://huggingface.co/prism-ml/Bonsai-8B-gguf): ```bash pip install -U "huggingface_hub[cli]" hf download prism-ml/Bonsai-8B-gguf --local-dir ./bonsai ``` 3. Run the same long prompt twice with a prompt cache, following the setup in the [first inference guide](https://yourwildcard.ai/docs/getting-started/first-inference.md): ```bash llama-cli -m ./bonsai/.gguf \ --prompt-cache cache.bin \ -p " Summarize the above." \ -n 64 2>&1 | tee run-01.log llama-cli -m ./bonsai/.gguf \ --prompt-cache cache.bin \ -p " List three facts from the above." \ -n 64 2>&1 | tee run-02.log ``` 4. Compare the prompt evaluation time printed in the two logs. Write down anything you cannot explain, e.g., why the second run only skips the tokens up to the first character that differs. 5. Reread the caching section with your questions in hand. Note what you now read differently, and compare your note against the sentence you wrote in step 1. If you prefer speculation over caching, run the demo in the [speculative decoding guide](https://yourwildcard.ai/docs/technical-guides/speculative-decoding.md) instead and watch the acceptance rate in the logs. ## Check yourself - **Name one claim from the material you misread before the run, and what the running system corrected.** Expected answer: any concrete pair works, e.g., "I read the cache as saving the whole prompt, and the second run showed it only reuses the prefix up to the first differing token." - **Why does reading the abstract before the run not spoil the loop?** Expected answer: the abstract tells you what the system claims, which you need to pick a demo. The mechanism, which is the part the run teaches, stays unread until you have questions. - **You finished a paper and have no questions. What does that signal, and what do you do?** Expected answer: it signals the reading never made contact with the system. Go run the smallest demo and collect outputs you cannot explain, then read again. ## Next steps - Work through the [first inference guide](https://yourwildcard.ai/docs/getting-started/first-inference.md) once, so future demos start from a known-good local setup. - Read the [speculative decoding guide](https://yourwildcard.ai/docs/technical-guides/speculative-decoding.md) after running its demo, in that order, and note the difference. - For the mechanism behind the acceptance rate arithmetic above, see [how speculative decoding works](/blog/how-speculative-decoding-works). When you can do this, you can execute an apply-first study loop and state the comprehension delta it produced. --- --- title: "Which parts of an LLM are safest to quantize: weights, activations, KV cache, or attention?" description: "Quantization risk runs from weights (safest) to activations, then KV cache, then attention. Treat them as four separate dials, not one switch." audience: inference-engineer pillar: foundational-concept book: inference-engineering chapter_ref: "Ch. 5, sec. 5.1.2 (pp. 126-127)" status: published last_reviewed: 2026-07-07 canonical_url: https://yourwildcard.ai/blog/safest-parts-of-llm-to-quantize --- # Which parts of an LLM are safest to quantize: weights, activations, KV cache, or attention? Quantization sensitivity forms a ladder from least to most risky: weights first, then activations, then KV cache, and attention last. Softmax is almost always left in original precision. Treat quantization as four separate dials with a default ordering, not one switch you flip for the whole model. ## Order the four targets by sensitivity Kiely's Inference Engineering (Ch. 5, sec. 5.1.2) lists the components of a model from least to most sensitive to quantization. Lowering the precision of a more sensitive component risks more quality loss for the same memory savings. 1. **Weights.** The linear layers are the least sensitive part of the model, so quantize them first. This is also where most of the memory is, so it is the biggest win. 2. **Activations.** The intermediate outputs of activation functions are only somewhat sensitive. The activation functions themselves are rarely quantized because they are a tiny fraction of the model. 3. **KV cache.** The cached keys and values from the attention calculation are moderately sensitive. Quantizing them frees memory for longer contexts and more concurrent requests. 4. **Attention.** The attention layers are highly sensitive, and Kiely singles out "equations like softmax" as the part to protect. All but the most aggressive schemes run softmax in the original precision. The ordering matters because it gives you a procedure. Start at weights, check quality, and only move down the list if you still need memory back. Do not turn all four dials at once, because then you cannot tell which one broke the output. ## Understand why errors compound at the bottom of the list Weight quantization introduces one rounding error per weight, and that error is applied the same way to every token. The model also has billions of weights, so no single weight carries much of the output. This is why weights tolerate 4-bit formats. The KV cache is different. The cached entry for each token is read by every later token in the sequence. A rounding error written into the cache at token 50 feeds into the attention result for token 51, and that result feeds into token 52. Kiely's point is that precision errors in the cache compound from token to token instead of staying local. Attention is the worst case for the same reason plus one more. Attention scores pass through softmax, which turns small input differences into large probability differences, so attention needs more dynamic range than a 4-bit or 8-bit format provides. Over a sequence of thousands of tokens, the book notes, these errors accumulate quickly. Even within a safe component you can be selective. Kiely notes that the input and output layers of the network are more sensitive than the middle layers, so quantization schemes often leave the first and last layers in original precision even when every other linear layer is 4-bit. ## Work through the KV cache arithmetic Here is what the KV cache dial buys you on a concrete model. Take an 8B model with 32 layers, 8 KV heads, and a head dimension of 128, e.g., [Bonsai 8B](https://huggingface.co/prism-ml/Bonsai-8B-gguf). Each token stores one key vector and one value vector per layer: ```text values per token = 2 x 32 layers x 8 heads x 128 dims = 65,536 FP16 (2 bytes each): 65,536 x 2 = 131,072 bytes = 128 KiB per token q8_0 (8.5 bits each): 65,536 x 1.0625 = 69,632 bytes = 68 KiB per token ``` At an 8,192-token context, the FP16 cache is 8,192 x 128 KiB, which is 1 GiB. The q8_0 cache is 544 MiB. One dial returned 480 MiB without touching the weights. These numbers are derived from the shapes above, not benchmarks, so your model's config may differ. The tradeoff is that the saving comes from the moderately risky rung. Kiely writes that a common production setup quantizes select linear layers, activations, and often the KV cache with a high dynamic range 8-bit format like FP8 or MXFP8, and leaves the attention layer alone even then. See the [KV cache guide](https://yourwildcard.ai/docs/technical-guides/kv-cache.md) for how the cache is laid out and why it grows with context length. ## Try it This takes about 20 minutes. You need llama.cpp and one GGUF model with 4-bit weights, e.g., the Q4_K_M file from [Bonsai 8B](https://huggingface.co/prism-ml/Bonsai-8B-gguf). No GPU is required. An 8B model at 4 bits runs on a laptop with about 8 GB of free RAM, and Apple silicon or a CUDA GPU just makes it faster. Put an article of a few hundred words in `article.txt` with a line like "Summarize this article in five bullet points:" at the top. Then run the same prompt with two KV cache settings: ```bash # Run 1: 4-bit weights, F16 KV cache (the default). llama-cli -m Bonsai-8B-Q4_K_M.gguf -f article.txt \ --temp 0 --seed 1 -n 256 > kv-f16.txt # Run 2: same everything, but the K half of the cache at q8_0. llama-cli -m Bonsai-8B-Q4_K_M.gguf -f article.txt \ --temp 0 --seed 1 -n 256 --cache-type-k q8_0 > kv-q8.txt diff kv-f16.txt kv-q8.txt ``` Temperature 0 and a fixed seed make the runs comparable, so any diff comes from the cache precision. On a short summarization prompt the outputs are often identical or differ in a few word choices, which is what "moderately sensitive" looks like in practice. If you want to push further, quantize the V half too with `--cache-type-v q8_0`, which in llama.cpp also requires the flash attention flag `-fa on`. ## Check yourself 1. Order the four quantization targets from safest to riskiest, and name the one part that is almost never quantized. Expected: weights, then activations, then KV cache, then attention, and softmax stays in original precision. 2. Why does a rounding error in the KV cache do more damage than the same error in a weight? Expected: a weight error is applied the same way to every token, but a cache error is read by every later token, so it feeds into each following attention result and compounds across the sequence. 3. Your 8B model fits in memory, but you cannot serve the context lengths you need. Which dial do you turn, and what do you check after? Expected: quantize the KV cache, e.g., to an 8-bit format, because context memory is cache memory. Then rerun a quality check on long prompts, since long sequences are where cache errors accumulate. 4. A teammate proposes shipping a build with weights, activations, cache, and attention all at 4-bit because it benchmarked fastest. What is wrong with that? Expected: it turned all four dials at once, including the riskiest one. Attention at 4 bits lacks the dynamic range softmax needs, and with every dial turned there is no way to isolate which change caused a quality drop. ## Next steps - [Quantization guide](https://yourwildcard.ai/docs/technical-guides/quantization.md) for number formats, granularity, and how to pick a precision per component. - [KV cache guide](https://yourwildcard.ai/docs/technical-guides/kv-cache.md) for how cache size scales with context and batch size. - [How do you verify a quantized model didn't get worse?](/blog/verify-quantized-model-quality) for the quality check you should run after each dial. When you can do this, you can order the four quantization targets by risk and apply them incrementally with a quality check at each step. --- --- title: "Should I use safetensors or ONNX for my model?" description: "Safetensors holds only tensor data, so it is safe and fast to load. ONNX bundles weights with a graph for portability. Many teams ship safetensors into vLLM." audience: inference-engineer pillar: ecosystem-player book: inference-engineering chapter_ref: "Ch. 4, secs. 4.2.2-4.2.3 Model File Formats" status: published last_reviewed: 2026-07-07 canonical_url: https://yourwildcard.ai/blog/safetensors-vs-onnx --- # Should I use safetensors or ONNX for my model? Safetensors stores only tensor data that a loader can memory map. It contains no executable code, so it is safer and faster to load. ONNX bundles the weights with an execution graph, so the same file runs anywhere an ONNX runtime exists. Many teams now skip ONNX export and ship safetensors straight into vLLM or TensorRT-LLM, especially for new architectures like multi-latent attention that make graph export painful. ## Separate the code from the data A model is two things. The weights are the numbers learned in training. The architecture is the code that says what to do with those numbers. The two formats split these differently. A safetensors file holds only the weights. Kiely's *Inference Engineering* (sec. 4.2.2) explains that the safety in the name comes from this restriction. An older generic format like a pickled `.bin` file can run arbitrary Python code while it loads, so a malicious model file can attack the machine that opens it. A safetensors file cannot do that, because there is nothing in it to execute. The architecture lives elsewhere, in a `config.json` file plus model code in a library or an inference engine. An ONNX file holds the weights and an execution graph together. The graph records every operation the model performs, so any ONNX runtime can run the model without knowing the architecture in advance. That is the portability case for ONNX, and Kiely calls it a good alternative when you want to store model graphs, not just weights. You may also see GGUF, a third format used by llama.cpp. Like ONNX it packs the weights and the metadata to run them into one file, e.g., the public [Bonsai 8B GGUF](https://huggingface.co/prism-ml/Bonsai-8B-gguf) release on Hugging Face. The same code and data question applies to it. ## Understand why export breaks on new architectures To make an ONNX file, you export the model from PyTorch. The exporter traces the model and rewrites every operation into the ONNX standard. The standard does not cover every PyTorch data structure, type, and operation, so export fails when the model uses something the standard cannot express. New architectures hit this first. Kiely's example (sec. 4.2.3) is DeepSeek V3, which introduced multi-latent attention. The PyTorch implementation of that attention variant is difficult to export, because the exporter has to rewrite it into standard operations that were not designed for it. The safetensors path avoids the problem. The weights are just tensors, and tensors always serialize. The architecture ships as code inside the inference engine instead, so supporting a new model means the vLLM or TensorRT-LLM maintainers write the model code once, and every user loads the same safetensors weights into it. This is why day zero support for a new model usually appears in the engines before any working ONNX export exists. Kiely writes that the industry "is bifurcating between the control of handwritten PyTorch code or the convenience of prebuilt inference engines" (sec. 4.2.3). In plain terms, both ends of that split load safetensors, and the ONNX step in the middle is the part teams drop. ONNX Runtime and TensorRT remain common where export is easy, and Kiely notes TensorRT is still strong for image and video models. ## Work through what loading actually costs Here is a worked example with derived numbers, not benchmarks. Take an 8B parameter model such as Bonsai 8B stored at 16-bit precision. Each parameter is 2 bytes, so the weights are 8 billion times 2 bytes, which is 16 GB. Hugging Face splits weights this large across several safetensors shards, e.g., four shards of about 4 GB each. Assume an SSD that reads at 3 GB/s. Reading 16 GB from disk takes 16 / 3, which is about 5.3 seconds, and any format pays that. The difference is what happens around the read. A pickle loader deserializes the whole file into CPU memory first, so you need roughly 16 GB of free RAM before the weights ever reach the GPU, and the unpickler runs code while it works. A safetensors loader memory maps the file instead. It reads a small header that lists each tensor's name, type, shape, and byte offset, then copies tensors to the GPU directly from the mapped file. You never need the full 16 GB in RAM at once, and nothing executes. Kiely's summary (sec. 4.2.2) is that memory mapping makes loading weights faster and safer. The header design is simple enough to inspect by hand. The first 8 bytes of a safetensors file are a little-endian integer giving the header length, and the header itself is plain JSON. The exercise below reads it. ## Try it Download a small model, inspect its safetensors shards and its `config.json`, and find where the architecture actually lives. This takes under 30 minutes, needs about 1 GB of disk, and needs no GPU. 1. Download a small public model. ```bash pip install -U huggingface_hub hf download HuggingFaceTB/SmolLM2-135M --local-dir ./smollm2 ls -lh ./smollm2 ``` 2. Read the safetensors header. This prints every tensor's name, shape, and byte offset. ```bash python3 -c " import json, struct with open('./smollm2/model.safetensors','rb') as f: n = struct.unpack('