Blog
Ternary Bonsai 27B is the main story here. Twenty-six focused posts connect the public checkpoint and whitepaper to local M4 Pro traces, runtime choices, tool calling, reasoning budgets, fine tuning, and the launch conversation. The broader inference archive follows below.
Ternary Bonsai 27B
Start with the field guide or the ecosystem roundup centered on The Information.
- What is Ternary Bonsai 27B?
- What Mac do you need for Ternary Bonsai 27B?
- How to run Ternary Bonsai 27B with MLX
- How to serve it through an OpenAI-compatible API
- How many bits per weight does it really use?
- Binary versus ternary weights
- Why the ideal and packaged model sizes differ
- How the KV cache changes memory use
- Measured M4 Pro performance
- What the quality benchmarks show
- Qwen3.6 27B versus Ternary Bonsai 27B
- Using Bonsai 27B with Hermes Agent
- Using Bonsai 27B in LM Studio
- Whether ODS is a tenable runtime path
- How to approach fine tuning
- Known limits
- The benchmark method
- Why reasoning-token budgets matter
- How prompt caching changes the result
- How to resume an interrupted download
- What we learned on a 24 GB Mac
- The structured tool-call test
- Why speed changes between runs
- How to compare vendor and local benchmarks
- How to verify the exact checkpoint
- What the ecosystem is saying about Bonsai 27B
Foundational concepts
The ideas under every inference decision, from the KV cache to quantization.
- Can 1-bit models do tool calling?
- Why does batch size trade latency for throughput in LLM serving?
- How much VRAM do I need to run an LLM?
- How does speculative decoding generate more than one token per forward pass?
- How does the KV cache make attention linear instead of quadratic?
- Why is model selection the biggest inference optimization?
- Why is LLM prefill compute-bound but decode memory-bound?
- Should I use a shared LLM API or a dedicated deployment?
- What are the three layers of an inference stack?
- How do I verify a quantized model hasn't lost quality?
- What are prefill and decode in LLM inference?
- What do TTFT and TPS actually measure for LLMs?
- What does 'intelligence density' actually measure — and why isn't it just tokens per second?
- Why do API providers charge less for cached input tokens?
- Why are GPUs faster than CPUs for AI inference?
- Why does quantization make LLM inference faster in both prefill and decode?
- Why should I report P99 latency instead of average?
- How should product teams think about accuracy vs. speed trade-offs in AI?
- What's the difference between fine-tuning and distillation?
- How many GPUs do I need to serve a large LLM?
- Why do images blow up my LLM's context window?
- Inference is fast but my app feels slow — what do I check?
- Is generative AI the same thing as AI?
- How do I make model cold starts faster?
- What metrics should I monitor for an LLM inference service?
- What's the difference between online and offline inference?
- How should I order my prompt to maximize prefix cache hits?
- Which parts of an LLM are safest to quantize: weights, activations, KV cache, or attention?
- Tensor vs pipeline vs expert parallelism: which should I use for LLM inference?
- Do inference optimizations like quantization and speculation stack together?
- Why are MoE models fast locally but not on servers?
- What determines whether speculative decoding actually speeds things up?
- What is disaggregated serving (prefill/decode separation) and when is it worth it?
- What is an ops:byte ratio and why does it matter for GPUs?
- What makes long context expensive, and what are chunked prefill and paged KV for?
- Why are floating-point formats better than integers for LLM quantization?
- Why is image generation compute-bound when the models are small?
Ecosystem players
The runtimes, engines, and platforms that serve small models, compared honestly.
- Does the 1-bit Bonsai whitepaper prove the efficiency thesis this wiki is built on?
- What do FlashAttention and PagedAttention actually optimize?
- How do I benchmark an LLM server properly?
- Ollama vs llama.cpp — which should I use for local inference?
- vLLM vs SGLang vs TensorRT-LLM — which inference engine should I pick?
- Why is Apple Silicon good for running LLMs locally?
- What's the fastest way to run Whisper locally — streaming or batch?
- Should I use safetensors or ONNX for my model?
- What are DeepSeek-R1 distilled models and should I run one locally?
- What is a MIG (multi-instance GPU) and when should I use one?
- What is GGUF block quantization and how do Q4_K models actually work?
- When is NVIDIA Dynamo worth the complexity?
- Who is your developer ecosystem actually for? End users vs ecosystem members
- What's the difference between benchmarking and profiling inference?
- What content do runtimes, MSPs, and inference studios need that end users never read?
- When should I use a draft model vs EAGLE vs n-gram speculation?
- Why should I pin exact dependency versions in my inference container?
- How do I serve embeddings for both bulk indexing and live search?
- What does 'sparse FLOPS' mean on a GPU spec sheet, and should I trust it?
- What is a CUDA kernel in simple terms?
- What is cache-aware routing and why does round-robin load balancing hurt LLM latency?
- Why do GPU kernels break when you upgrade GPUs?
- Why don't prefix caching and tensor parallelism help embedding models?
- Why do multi-GPU LLM deployments need NVLink or InfiniBand?
End-user case studies
How real teams in regulated and cost-sensitive settings adopt local AI.
- What does a hospital CTO actually ask before running a model on-prem?
- Should I run AI on-device or in the cloud? The eight-factor tradeoff
- At what token volume does running my own GPU beat a per-token API?
- What does an AI product manager actually do day to day?
- How many clicks to first inference? Auditing your docs like a funnel
- When is GenAI the wrong tool for personalization?
- What belongs in a local-AI runbook for a compliance-bound team?
- Should I use a speech-to-speech model or an ASR-LLM-TTS pipeline?
- How can I transcribe an hour of audio in seconds?
- When is scale-to-zero a bad idea for AI apps?
- Why do worked examples beat case studies in regulated industries?
- What's the difference between AGI hype and today's AI capabilities?
- Are AI agents just chatbots with tools?
- What is speech recognition vs. text-to-speech, in plain terms?
- How do you benchmark a local model when your budget is a power outlet, not a GPU cluster?
- Do I need an LLM for ticket routing, or is a classifier enough?
- How do I compare per-token API costs to renting my own GPUs?
- Why do data-rich companies have an AI advantage?
- When should I use a few-step image model?
- Is a bigger model ever cheaper? Rethinking cost per unit of capability
- Should I use one omni-modal model or a pipeline of small specialists?
- Should you run the demo before you finish the paper?
- How do safety frameworks translate into an on-prem AI deployment checklist?
- Why should every demo be extendable, not just runnable?
- What if your UI taught users how inference works?
- What legitimate uses do deepfakes have, and how should teams manage the risk?
- Why do voice assistants struggle with accents, and whose job is it to fix?
- Will every product manager become an AI product manager?