> Full documentation content: https://yourwildcard.ai/llms-full.txt # PrismML Developer Documentation > Community documentation for PrismML's Bonsai model family (1-bit Bonsai 8B, ternary variants, Bonsai Image 4B) and the local-inference supply chain that runs them. Every page is also available as raw Markdown by appending .md to its URL. - [1-bit Bonsai 8B whitepaper (PDF)](https://yourwildcard.ai/whitepaper/1-bit-bonsai-8b-whitepaper.pdf): Primary source for architecture, benchmark, throughput, and energy claims. - [Bonsai 8B GGUF weights](https://huggingface.co/prism-ml/Bonsai-8B-gguf): Official 1.16 GB quantized weights for llama.cpp-compatible runtimes, Apache 2.0. ## Start Here - [Orientation](https://yourwildcard.ai/docs/start-here/orientation.md): PrismML is a family of small open-weight language models built to run on ordinary hardware such as laptops, phones, and efficient servers. - [Audience and standards](https://yourwildcard.ai/docs/start-here/audience-and-standards.md): This wiki explains the tools that take a small open-weight language model from downloaded weights to a running product, and it is written for people who build and explain AI products. - [Wiki contract](https://yourwildcard.ai/docs/start-here/wiki-contract.md): The wiki contract is the checklist a maintainer uses to decide whether a page in this wiki is finished. - [Use these docs with AI](https://yourwildcard.ai/docs/start-here/use-with-ai.md): These docs are built for coding agents as much as for people. ## Learning Paths - [Inference The Hard Way](https://yourwildcard.ai/docs/learning-paths/inference-the-hard-way.md): Inference The Hard Way is a lab course that shows how a language model goes from bytes on disk to tokens on screen. - [Kubernetes The Hard Way](https://yourwildcard.ai/docs/learning-paths/kubernetes-the-hard-way.md): Kubernetes The Hard Way is a free course that walks you through building a working Kubernetes cluster by hand, one component at a time, from machines and certificates through etcd, the control plane, workers, networking, and smoke tests. - [Designing the docs chat](https://yourwildcard.ai/docs/learning-paths/designing-the-docs-chat.md): The docs chat agent is a worked example of the two Hard Way courses: Inference The Hard Way chose the model and the prompt shape, and Kubernetes The Hard Way ruled out every deployment except the browser. ## PrismML - [Bonsai 27B field guide](https://yourwildcard.ai/docs/prismml/bonsai-27b.md): Ternary Bonsai 27B is the main model covered by these docs. - [Model family](https://yourwildcard.ai/docs/prismml/model-family.md): PrismML is a company that builds small compressed language and image models, released under the Bonsai name, that you can run on a laptop, a phone, or in a browser. - [Whitepaper benchmarks](https://yourwildcard.ai/docs/prismml/whitepaper-benchmarks.md): The 1-bit Bonsai 8B whitepaper reports how a 1.15 GB model scores against full-precision models ten times its size, and how fast and efficiently it runs on real hardware. - [Claim and source matrix](https://yourwildcard.ai/docs/prismml/claim-source-matrix.md): The claim source matrix is the list of facts we publish about PrismML models, each traced to a named source with a verification status. ## Ecosystem Map - [Supply-chain layers](https://yourwildcard.ai/docs/ecosystem/supply-chain-layers.md): The ML supply chain is the chain of tools and decisions that take a small model from raw data to a product running on a real device. - [llama.cpp](https://yourwildcard.ai/docs/ecosystem/llama-cpp.md): llama.cpp is an open source C/C++ runtime that runs GGUF model files on your own computer, on CPU, NVIDIA CUDA, or Apple Metal. - [MLX](https://yourwildcard.ai/docs/ecosystem/mlx.md): MLX is Apple's open source machine learning framework for Apple Silicon, and it is the native way to run PrismML Bonsai models on a Mac. - [Ollama](https://yourwildcard.ai/docs/ecosystem/ollama.md): Ollama is a tool that downloads, runs, and serves language models on your own laptop or workstation with one command. - [LM Studio](https://yourwildcard.ai/docs/ecosystem/lm-studio.md): LM Studio is a desktop app that downloads and runs language models on your own machine through a graphical interface, with no command line. - [vLLM](https://yourwildcard.ai/docs/ecosystem/vllm.md): vLLM is an open source engine that serves a language model checkpoint as a high-throughput, OpenAI-compatible API. - [SGLang](https://yourwildcard.ai/docs/ecosystem/sglang.md): SGLang is an open source server for running large language models on GPUs at high throughput, built by the LMSYS team at Berkeley. - [Ray and Anyscale](https://yourwildcard.ai/docs/ecosystem/ray.md): Ray is an open source Python framework for running data processing, model training, and model serving across many machines. - [Unsloth](https://yourwildcard.ai/docs/ecosystem/unsloth.md): Unsloth is a toolkit for fine-tuning language models, which means training an existing model further on your own data so it behaves the way you want. - [Liquid AI](https://yourwildcard.ai/docs/ecosystem/liquid-ai.md): Liquid Foundation Models (LFMs) are a family of small language models from Liquid AI that do not use the standard transformer design, and Liquid builds them to run on small devices such as laptops and phones. ## Build And Run - [Runtime map](https://yourwildcard.ai/docs/build-and-run/runtime-map.md): A runtime is the program that loads a PrismML model's weights and generates text on your hardware, e.g., llama.cpp on a laptop. - [Bonsai with llama.cpp](https://yourwildcard.ai/docs/build-and-run/bonsai-llamacpp.md): Bonsai is PrismML's 8B open-weight language model, and llama.cpp is a C++ runtime that runs models like it on a laptop or a single GPU. - [Bonsai with MLX](https://yourwildcard.ai/docs/build-and-run/bonsai-mlx.md): MLX is Apple's machine learning framework for Apple Silicon Macs, and Bonsai 8B is PrismML's small open-weight language model. - [Bonsai with Ollama](https://yourwildcard.ai/docs/build-and-run/bonsai-ollama.md): Ollama is a free desktop tool that runs open-weight language models on your own machine, and Bonsai 8B is PrismML's small 1-bit quantized model. - [Bonsai with LM Studio](https://yourwildcard.ai/docs/build-and-run/bonsai-lmstudio.md): LM Studio is a free desktop app that downloads and runs local language models behind a graphical chat interface, with no command line required. - [Bonsai Image 4B local](https://yourwildcard.ai/docs/build-and-run/bonsai-image.md): Bonsai Image 4B is PrismML's small text-to-image model, released in 1-bit and ternary variants under the Apache 2.0 license. - [Bonsai Image in the browser](https://yourwildcard.ai/docs/build-and-run/bonsai-image-webgpu.md): Bonsai Image 4B is PrismML's small image generation model, and it can run entirely inside a WebGPU browser with no server. - [Bonsai Studio on iPhone](https://yourwildcard.ai/docs/build-and-run/bonsai-studio-ios.md): Bonsai Studio is PrismML's iPhone app for Bonsai Image 4B, a small open-weight image model that generates images on the phone itself. - [Ask the docs chat agent](https://yourwildcard.ai/docs/build-and-run/docs-chat.md): The docs chat agent is 1-bit Bonsai 1.7B answering questions about this documentation, running entirely in your browser over WebGPU. ## Technical Guides - [KV cache](https://yourwildcard.ai/docs/technical-guides/kv-cache.md): The KV cache is the block of memory where a language model stores the keys and values it has already computed for past tokens, so it does not recompute them for every new token. - [Prefill vs decode](https://yourwildcard.ai/docs/technical-guides/prefill-vs-decode.md): Prefill and decode are the two phases of running a language model. - [Weights on disk](https://yourwildcard.ai/docs/technical-guides/weights-on-disk.md): Quantization means storing a model's weights with fewer bits per weight than the precision they were trained in, so the file on disk gets smaller. - [Kernel reading](https://yourwildcard.ai/docs/technical-guides/kernel-reading.md): Kernel reading means opening the runtime source code and tracing what a compute kernel actually does with a model's weights. - [Bandwidth ledger](https://yourwildcard.ai/docs/technical-guides/bandwidth-ledger.md): A bandwidth ledger is a record that puts a predicted decode speed next to a measured one, so you can see how close a device gets to its physical limit. - [Architectures beyond transformers](https://yourwildcard.ai/docs/technical-guides/architectures.md): State space models and hybrid designs are alternatives to the transformer that trade some recall ability for much lower compute and memory at long context. - [Feature stores](https://yourwildcard.ai/docs/technical-guides/feature-stores.md): A feature store is a system that computes a data value once, stores it with a timestamp and a key, and serves that same value to both your training jobs and your live agent. - [Training and alignment](https://yourwildcard.ai/docs/technical-guides/training-and-alignment.md): Fine-tuning updates a pretrained model on your own data, and preference alignment then steers the model toward outputs people prefer. - [Evaluation concepts](https://yourwildcard.ai/docs/technical-guides/evaluation-concepts.md): Evaluation is how you check that a model still does its job on the device where it will actually run, not just on a cloud leaderboard. ## Benchmarks - [Evaluation harness](https://yourwildcard.ai/docs/benchmarks/evaluation-harness.md): The evaluation harness is a fixed protocol for measuring how a model actually behaves on a real device. - [Device report schema](https://yourwildcard.ai/docs/benchmarks/device-report-schema.md): A device report is a record of one benchmark run. ## Sources And Governance - [Source policy](https://yourwildcard.ai/docs/sources/source-policy.md): A source is any document, repository, post, or recording that a fact on this wiki comes from, and not every source is strong enough to support a published fact. - [X discovery notes](https://yourwildcard.ai/docs/sources/x-discovery-notes.md): An X post about PrismML is a lead, not a fact. - [Operating model](https://yourwildcard.ai/docs/governance/operating-model.md): The operating model is the set of roles, steps, and schedules that keep this wiki accurate. - [DevRel playbooks](https://yourwildcard.ai/docs/governance/devrel-playbooks.md): A DevRel playbook is a short, repeatable procedure for a recurring community task, such as announcing a release or preparing a live demo. - [Page types and templates](https://yourwildcard.ai/docs/governance/page-types-and-templates.md): Every page in this wiki is one of seven page types, and each type comes with a fixed front matter block and a template. - [Evaluation rubric](https://yourwildcard.ai/docs/governance/evaluation-rubric.md): The evaluation rubric is the scoring standard that decides whether a page in this wiki is ready to publish. ## Blog - [All posts](https://yourwildcard.ai/blog.md): Ternary Bonsai 27B is the main story here. ## Contribute - [Contributor guide](https://yourwildcard.ai/docs/contribute/contributor-guide.md): A contribution to this wiki is a documented fact about small local models or the tools that run them, backed by a source a reviewer can check. - [Contribution backlog](https://yourwildcard.ai/docs/contribute/backlog.md): The contribution backlog is the ordered list of work this wiki needs next, from hardware verification to new runtime recipes. ## Blog - [Blog index](https://yourwildcard.ai/blog.md): Ternary Bonsai 27B is the main story here. - [How should product teams think about accuracy vs. speed trade-offs in AI?](https://yourwildcard.ai/blog/accuracy-vs-speed-tradeoffs-ai-products.md): Accuracy versus speed is a product decision expressed through technical dials, chiefly model size and quantization level. - [What's the difference between AGI hype and today's AI capabilities?](https://yourwildcard.ai/blog/agi-hype-vs-shipping-reality.md): The difference is shipped narrow capability against speculative general capability. - [Are AI agents just chatbots with tools?](https://yourwildcard.ai/blog/are-ai-agents-just-chatbots-with-tools.md): No. - [What is speech recognition vs. text-to-speech, in plain terms?](https://yourwildcard.ai/blog/asr-tts-plain-terms-offline-voice-assistant.md): Speech recognition turns human speech into text. - [Why does batch size trade latency for throughput in LLM serving?](https://yourwildcard.ai/blog/batch-size-latency-throughput-tradeoff.md): Bigger batches raise total tokens per second by keeping the GPU busy during memory-bound decode, but each individual user's stream slows down. - [How do you benchmark a local model when your budget is a power outlet, not a GPU cluster?](https://yourwildcard.ai/blog/benchmarking-on-a-power-outlet-budget.md): When watts and RAM are the hard constraints, benchmark inside the envelope. - [What's the difference between benchmarking and profiling inference?](https://yourwildcard.ai/blog/benchmarking-vs-profiling-inference.md): Benchmarking tells you how the system performs. - [Binary versus Ternary Bonsai 27B](https://yourwildcard.ai/blog/binary-vs-ternary-bonsai-27b.md): Use Ternary Bonsai 27B when quality is the main goal and a laptop has enough memory. - [What the ecosystem is saying about Bonsai 27B](https://yourwildcard.ai/blog/bonsai-27b-ecosystem-reactions.md): The main public story has two parts. - [Does the 1-bit Bonsai whitepaper prove the efficiency thesis this wiki is built on?](https://yourwildcard.ai/blog/bonsai-whitepaper-and-the-efficiency-thesis.md): This wiki is organized around two arguments about where AI is heading. - [Can 1-bit models do tool calling?](https://yourwildcard.ai/blog/can-1-bit-models-do-tool-calling.md): Yes, but not unconstrained. - [Do I need an LLM for ticket routing, or is a classifier enough?](https://yourwildcard.ai/blog/classifier-or-llm-for-ticket-routing.md): Usually a classifier is enough. - [How many clicks to first inference? Auditing your docs like a funnel](https://yourwildcard.ai/blog/clicks-to-first-inference-docs-funnel.md): Treat your quickstart as a conversion funnel: count every click, copy-paste, and decision between landing and a first successful inference, then cut the count like you would cart abandonment. - [How do I compare per-token API costs to renting my own GPUs?](https://yourwildcard.ai/blog/compare-api-costs-to-renting-gpus.md): Do not reverse engineer a per-token price from GPU rates. - [What content do runtimes, MSPs, and inference studios need that end users never read?](https://yourwildcard.ai/blog/content-for-runtimes-msps-inference-studios.md): Ecosystem members build on your project, so they read the integration surface, e.g., API stability guarantees, extension points, version support matrices, and packaging conventions. - [Why do data-rich companies have an AI advantage?](https://yourwildcard.ai/blog/data-advantage-without-an-llm.md): Data-rich companies have an AI advantage because models commoditize and datasets do not. - [Do inference optimizations like quantization and speculation stack together?](https://yourwildcard.ai/blog/do-inference-optimizations-stack.md): Not automatically. - [When should I use a draft model vs EAGLE vs n-gram speculation?](https://yourwildcard.ai/blog/draft-model-vs-eagle-vs-ngram-speculation.md): Draft-target speculation is the easiest to set up but has the highest overhead. - [What's the fastest way to run Whisper locally — streaming or batch?](https://yourwildcard.ai/blog/fastest-way-to-run-whisper-locally.md): It depends on which speed you are buying. - [When should I use a few-step image model?](https://yourwildcard.ai/blog/few-step-image-models-for-realtime-features.md): Few-step image models generate in eight or fewer denoising steps. - [How to fine tune Ternary Bonsai 27B](https://yourwildcard.ai/blog/fine-tune-ternary-bonsai-27b.md): Start with a LoRA or QLoRA adapter instead of changing all 27 billion ternary weights. - [What's the difference between fine-tuning and distillation?](https://yourwildcard.ai/blog/fine-tuning-vs-distillation.md): Fine-tuning adapts a model to a domain with new training data. - [What do FlashAttention and PagedAttention actually optimize?](https://yourwildcard.ai/blog/flashattention-vs-pagedattention.md): FlashAttention and PagedAttention are lossless implementation work, not new math. - [When is GenAI the wrong tool for personalization?](https://yourwildcard.ai/blog/genai-wrong-tool-for-personalization.md): GenAI is the wrong tool when the job is ranking, because recommendation is largely traditional AI. - [What does a hospital CTO actually ask before running a model on-prem?](https://yourwildcard.ai/blog/hospital-cto-on-prem-ai-questions.md): The order of evaluation is the surprise. - [How many GPUs do I need to serve a large LLM?](https://yourwildcard.ai/blog/how-many-gpus-to-serve-an-llm.md): Minimum GPU count is arithmetic you can do before touching hardware. - [How much VRAM do I need to run an LLM?](https://yourwildcard.ai/blog/how-much-vram-to-run-an-llm.md): VRAM must hold the model weights plus roughly 50 percent headroom for the KV cache and activations, so weights fitting is not enough. - [How does speculative decoding generate more than one token per forward pass?](https://yourwildcard.ai/blog/how-speculative-decoding-works.md): Decode is limited by memory bandwidth, which leaves compute sitting idle. - [How do I benchmark an LLM server properly?](https://yourwildcard.ai/blog/how-to-benchmark-an-llm-server.md): A good benchmark mirrors production on four axes: sequence lengths, traffic volume and pattern, request contents, and sampling parameters. - [Why do images blow up my LLM's context window?](https://yourwildcard.ai/blog/images-blow-up-context-window.md): A single high-resolution image adds roughly a thousand visual tokens to the input sequence. - [Inference is fast but my app feels slow — what do I check?](https://yourwildcard.ai/blog/inference-fast-but-app-feels-slow.md): Track two numbers. - [Is a bigger model ever cheaper? Rethinking cost per unit of capability](https://yourwildcard.ai/blog/is-a-bigger-model-ever-cheaper.md): Occasionally yes. - [Is generative AI the same thing as AI?](https://yourwildcard.ai/blog/is-generative-ai-the-same-as-ai.md): No. - [How does the KV cache make attention linear instead of quadratic?](https://yourwildcard.ai/blog/kv-cache-linear-attention.md): Attention is quadratic in sequence length, but the KV cache stores key/value pairs for every prior token so each decode step only computes attention for the new token, which is linear time. - [What belongs in a local-AI runbook for a compliance-bound team?](https://yourwildcard.ai/blog/local-ai-runbook-for-compliance-teams.md): A runbook an end user can hand to auditors covers four things demos never do: rollback, which is how to return to the last known-good model, model provenance, which is what weights came from where and how you verified them, failure modes, which is what breaks and what users see, and operational boundaries, which is who can change what. - [How do I make model cold starts faster?](https://yourwildcard.ai/blog/make-model-cold-starts-faster.md): Cold start time splits into four phases you optimize separately: GPU procurement, image load, weight load, and engine startup. - [What metrics should I monitor for an LLM inference service?](https://yourwildcard.ai/blog/metrics-to-monitor-llm-inference.md): Monitor seven families together: volume, request and response sizes, response codes, latency percentiles (TTFT, TPS, end-to-end), replica count, utilization, and queue depth. - [Why is model selection the biggest inference optimization?](https://yourwildcard.ai/blog/model-selection-biggest-inference-optimization.md): All else equal, smaller models are always faster and cheaper, so choosing the smallest model that passes your evals beats any runtime trick. - [Why are MoE models fast locally but not on servers?](https://yourwildcard.ai/blog/moe-fast-locally-slow-on-servers.md): MoE models activate only a fraction of their parameters per token, e.g., 22B of 235B, which makes single-request local inference cheap. - [Should I use one omni-modal model or a pipeline of small specialists?](https://yourwildcard.ai/blog/ocr-model-vs-vlm-for-receipts.md): Omni models are convenient, but small specialists often beat the same capability baked into a VLM at a fraction of the size. - [Ollama vs llama.cpp — which should I use for local inference?](https://yourwildcard.ai/blog/ollama-vs-llama-cpp.md): Ollama and llama.cpp sit on different layers of the same stack, so this is not a fair fight between two rivals. - [Should I run AI on-device or in the cloud? The eight-factor tradeoff](https://yourwildcard.ai/blog/on-device-vs-cloud-ai-inference.md): Local inference wins on four factors: latency, privacy, offline independence, and zero marginal cost. - [What's the difference between online and offline inference?](https://yourwildcard.ai/blog/online-vs-offline-inference.md): Online inference optimizes for latency because a user is waiting. - [Why should I pin exact dependency versions in my inference container?](https://yourwildcard.ai/blog/pin-dependency-versions-inference-containers.md): Inference dependency chains, from the CUDA toolkit to torch to the engine build, are long and fragile, and the ecosystem breaks often. - [Why is LLM prefill compute-bound but decode memory-bound?](https://yourwildcard.ai/blog/prefill-compute-bound-decode-memory-bound.md): Prefill loads the model weights once and multiplies large matrices, so it performs many operations for every byte it reads from memory. - [How should I order my prompt to maximize prefix cache hits?](https://yourwildcard.ai/blog/prompt-ordering-for-prefix-cache-hits.md): The cache match runs from the first token until the first non-repeated token. - [Qwen3.6 27B versus Ternary Bonsai 27B](https://yourwildcard.ai/blog/qwen36-27b-vs-ternary-bonsai-27b.md): Qwen3.6 27B is the base multimodal model. - [How to run Ternary Bonsai 27B with MLX](https://yourwildcard.ai/blog/run-ternary-bonsai-27b-mlx.md): Install the current mlx-lm package, then pass the Hugging Face model ID to mlx_lm.generate or mlx_lm.server. - [Should you run the demo before you finish the paper?](https://yourwildcard.ai/blog/run-the-demo-before-finishing-the-paper.md): For a systems paper, yes. - [Which parts of an LLM are safest to quantize: weights, activations, KV cache, or attention?](https://yourwildcard.ai/blog/safest-parts-of-llm-to-quantize.md): Quantization sensitivity forms a ladder from least to most risky: weights first, then activations, then KV cache, and attention last. - [Should I use safetensors or ONNX for my model?](https://yourwildcard.ai/blog/safetensors-vs-onnx.md): Safetensors stores only tensor data that a loader can memory map. - [How do safety frameworks translate into an on-prem AI deployment checklist?](https://yourwildcard.ai/blog/safety-framework-to-onprem-checklist.md): Framework language becomes deployable when every clause maps to a checkable item on one concrete deployment. - [How do I serve embeddings for both bulk indexing and live search?](https://yourwildcard.ai/blog/serving-embeddings-bulk-vs-live.md): Embedding traffic splits into high-throughput backfills and low-latency lookups, and one deployment tuned for both satisfies neither. - [Should I use a shared LLM API or a dedicated deployment?](https://yourwildcard.ai/blog/shared-llm-api-vs-dedicated-deployment.md): Start on pay-per-token shared APIs. - [Why should every demo be extendable, not just runnable?](https://yourwildcard.ai/blog/should-every-demo-be-extendable.md): A runnable demo proves your software works. - [What does 'sparse FLOPS' mean on a GPU spec sheet, and should I trust it?](https://yourwildcard.ai/blog/should-i-trust-sparse-flops.md): Vendors advertise sparse FLOPS measured with 2:4 structured sparsity, and those numbers run roughly double the dense figures. - [Should I use a speech-to-speech model or an ASR-LLM-TTS pipeline?](https://yourwildcard.ai/blog/speech-to-speech-vs-asr-llm-tts-pipeline.md): Use the pipeline. - [Tensor vs pipeline vs expert parallelism: which should I use for LLM inference?](https://yourwildcard.ai/blog/tensor-vs-pipeline-vs-expert-parallelism.md): Tensor parallelism is the low-latency default within a node. - [What we learned running Bonsai 27B on a 24 GB Mac](https://yourwildcard.ai/blog/ternary-bonsai-27b-24gb-memory.md): Yes for moderate text work with one loaded runtime. - [How we benchmarked Ternary Bonsai 27B](https://yourwildcard.ai/blog/ternary-bonsai-27b-benchmark-method.md): We loaded the exact PrismML MLX checkpoint in LM Studio 0.4.19 on a 24 GB M4 Pro Mac. - [Why Ternary Bonsai 27B is called a 1.58 bit model](https://yourwildcard.ai/blog/ternary-bonsai-27b-bits-per-weight.md): A ternary weight has three possible values. - [How to resume the Ternary Bonsai 27B download](https://yourwildcard.ai/blog/ternary-bonsai-27b-download-resume.md): Keep one download owner, confirm which cache it uses, and resume into that same Hugging Face cache. - [How to use Ternary Bonsai 27B with Hermes Agent](https://yourwildcard.ai/blog/ternary-bonsai-27b-hermes-agent.md): PrismML documents Hermes Agent as a supported client for its OpenAI compatible server. - [How the KV cache affects Ternary Bonsai 27B memory](https://yourwildcard.ai/blog/ternary-bonsai-27b-kv-cache.md): Qwen3.6 27B uses hybrid attention, so only 16 of its 64 layers grow a normal full attention KV cache. - [Ternary Bonsai 27B limitations and verification checklist](https://yourwildcard.ai/blog/ternary-bonsai-27b-limitations.md): Ternary Bonsai 27B does not match full precision quality on every task, current ternary files are larger than the 5.9 GB ideal, GGUF needs PrismML's ternary runtime path, and speculative decoding is not yet a default speed win on Apple silicon. - [How to add Ternary Bonsai 27B to LM Studio](https://yourwildcard.ai/blog/ternary-bonsai-27b-lm-studio.md): LM Studio 0.4.19 loaded and generated from the exact PrismML MLX checkpoint as a Qwen3.5 family, 27B, 2 bit model. - [What Mac do you need for Ternary Bonsai 27B?](https://yourwildcard.ai/blog/ternary-bonsai-27b-mac-requirements.md): Use an Apple silicon Mac with enough free unified memory for the 8.49 GB MLX bundle, runtime buffers, the operating system, and the growing context cache. - [How to verify the exact Ternary Bonsai 27B model](https://yourwildcard.ai/blog/ternary-bonsai-27b-model-provenance.md): Record the Hugging Face repository, immutable revision, exact weight file size, and SHA 256 hash. - [How large is Ternary Bonsai 27B?](https://yourwildcard.ai/blog/ternary-bonsai-27b-model-size.md): PrismML reports three useful size figures. - [How Ternary Bonsai 27B fits into ODS](https://yourwildcard.ai/blog/ternary-bonsai-27b-ods.md): ODS is a tenable application stack for Ternary Bonsai 27B, but its normal macOS model path assumes a native llama-server and a GGUF model. - [How to serve Ternary Bonsai 27B through an OpenAI compatible API](https://yourwildcard.ai/blog/ternary-bonsai-27b-openai-api.md): Start `mlx_lm.server` with the PrismML model ID, bind it to localhost, and point your client at `http://127.0.0.1:8080/v1`. - [Ternary Bonsai 27B performance on an M4 Pro Mac](https://yourwildcard.ai/blog/ternary-bonsai-27b-performance-m4-pro.md): PrismML reports 18.0 tokens per second for 128 token generation and 125 tokens per second for 512 token prompt processing on an M4 Pro using llama.cpp Metal. - [How prompt caching changes Bonsai 27B benchmarks](https://yourwildcard.ai/blog/ternary-bonsai-27b-prompt-cache.md): Yes. - [How good is Ternary Bonsai 27B?](https://yourwildcard.ai/blog/ternary-bonsai-27b-quality-benchmarks.md): PrismML reports an 80.49 average across 15 thinking mode benchmarks for Ternary Bonsai 27B. - [Ternary Bonsai 27B needs room for reasoning tokens](https://yourwildcard.ai/blog/ternary-bonsai-27b-reasoning-token-budget.md): The model can use the whole output budget for reasoning before it writes a final answer. - [Why Bonsai 27B speed changes between runs](https://yourwildcard.ai/blog/ternary-bonsai-27b-runtime-variance.md): Context size, prompt cache state, model reloads, and system memory pressure can all change a local result. - [Ternary Bonsai 27B passed a structured tool call](https://yourwildcard.ai/blog/ternary-bonsai-27b-tool-calling-test.md): Yes in our basic local check. - [How to compare PrismML benchmarks with local Bonsai 27B tests](https://yourwildcard.ai/blog/ternary-bonsai-27b-vendor-vs-local-benchmarks.md): Only with clear limits. - [What are the three layers of an inference stack?](https://yourwildcard.ai/blog/three-layers-of-an-inference-stack.md): Production inference is three cooperating layers, and not only GPUs plus weights. - [At what token volume does running my own GPU beat a per-token API?](https://yourwildcard.ai/blog/token-volume-where-own-gpu-beats-api.md): There is a computable crossover. - [How can I transcribe an hour of audio in seconds?](https://yourwildcard.ai/blog/transcribe-an-hour-of-audio-in-seconds.md): With a two-stage pipeline, not a faster model. - [What if your UI taught users how inference works?](https://yourwildcard.ai/blog/ui-that-teaches-how-inference-works.md): Show retrieval, prefill, and decode as separate visible stages in your chat window. - [How do I verify a quantized model hasn't lost quality?](https://yourwildcard.ai/blog/verify-quantized-model-quality.md): Production-ready quantization means zero perceptible quality loss, and you verify that three ways: a perplexity comparison, a standard benchmark, and an evaluation on your own domain prompts. - [vLLM vs SGLang vs TensorRT-LLM — which inference engine should I pick?](https://yourwildcard.ai/blog/vllm-vs-sglang-vs-tensorrt-llm.md): vLLM and SGLang are easy to run, with day zero support for new models and broad hardware coverage. - [What legitimate uses do deepfakes have, and how should teams manage the risk?](https://yourwildcard.ai/blog/voice-cloning-consent-and-watermarking.md): Legitimate uses of voice cloning include accessibility voices for people who lost speech, localization and dubbing, and consented brand voices. - [What are DeepSeek-R1 distilled models and should I run one locally?](https://yourwildcard.ai/blog/what-are-deepseek-r1-distilled-models.md): They are small Llama and Qwen models trained as students on R1's output distributions, and the architecture choice is the strategic part. - [What are prefill and decode in LLM inference?](https://yourwildcard.ai/blog/what-are-prefill-and-decode.md): LLM inference has two phases. - [What determines whether speculative decoding actually speeds things up?](https://yourwildcard.ai/blog/what-determines-speculative-decoding-speedup.md): Three factors set the speedup: draft token cost, draft sequence length, and token acceptance rate. - [What do TTFT and TPS actually measure for LLMs?](https://yourwildcard.ai/blog/what-do-ttft-and-tps-measure.md): TTFT measures the compute-bound prefill phase, and TPS measures the bandwidth-bound decode phase. - [What does an AI product manager actually do day to day?](https://yourwildcard.ai/blog/what-does-an-ai-product-manager-do.md): The AI product manager turns a model capability into a product users trust, and the hard part is rarely the model. - [What is a CUDA kernel in simple terms?](https://yourwildcard.ai/blog/what-is-a-cuda-kernel.md): A CUDA kernel is a function of parallel code that runs on an NVIDIA GPU, with one thread typically handling one data element. - [What is a MIG (multi-instance GPU) and when should I use one?](https://yourwildcard.ai/blog/what-is-a-mig-fractional-gpu.md): Big GPUs like the H100 can split into up to seven fractional instances, and serving a 1 to 3B model on a whole H100 wastes money. - [What is cache-aware routing and why does round-robin load balancing hurt LLM latency?](https://yourwildcard.ai/blog/what-is-cache-aware-routing.md): With multiple replicas, routing purely by load throws away prefix caches. - [What is disaggregated serving (prefill/decode separation) and when is it worth it?](https://yourwildcard.ai/blog/what-is-disaggregated-serving.md): Disaggregation runs compute-bound prefill and memory-bound decode on separate engines connected by KV cache transfer, letting each pool scale to its own bottleneck. - [What is GGUF block quantization and how do Q4_K models actually work?](https://yourwildcard.ai/blog/what-is-gguf-block-quantization.md): A scale factor can apply to a whole tensor, to one channel, or to a small block of values. - [What does 'intelligence density' actually measure — and why isn't it just tokens per second?](https://yourwildcard.ai/blog/what-is-intelligence-density.md): Intelligence density measures usable capability per GB of memory, per watt, and per dollar, not raw speed or parameter count. - [What is an ops:byte ratio and why does it matter for GPUs?](https://yourwildcard.ai/blog/what-is-ops-byte-ratio.md): Every GPU has an ops:byte ratio, which is its peak FLOPS divided by its memory bandwidth, about 295 for an H100 in FP16. - [What is Ternary Bonsai 27B?](https://yourwildcard.ai/blog/what-is-ternary-bonsai-27b.md): Ternary Bonsai 27B is PrismML's low bit version of Qwen3.6 27B. - [What makes long context expensive, and what are chunked prefill and paged KV for?](https://yourwildcard.ai/blog/what-makes-long-context-expensive.md): KV cache memory grows linearly with sequence length until attention is your top VRAM consumer. - [When is NVIDIA Dynamo worth the complexity?](https://yourwildcard.ai/blog/when-is-nvidia-dynamo-worth-it.md): Dynamo adds KV-aware routing, prefill and decode disaggregation, and multi-node parallelism on top of any inference engine. - [When is scale-to-zero a bad idea for AI apps?](https://yourwildcard.ai/blog/when-scale-to-zero-is-a-bad-idea.md): Scale-to-zero needs fast cold starts plus robust queueing, and it suits dev environments and periodic workloads. - [Who is your developer ecosystem actually for? End users vs ecosystem members](https://yourwildcard.ai/blog/who-is-your-developer-ecosystem-for.md): CNCF-style segmentation splits an ecosystem into two groups. - [Why is Apple Silicon good for running LLMs locally?](https://yourwildcard.ai/blog/why-apple-silicon-runs-llms-well.md): Apple's unified memory gives the GPU access to far more memory than consumer NVIDIA cards, up to 512 GB versus 32 GB at the high end, at lower bandwidth. - [Why do API providers charge less for cached input tokens?](https://yourwildcard.ai/blog/why-cached-input-tokens-cost-less.md): Prefix caching reuses the KV cache of a shared prompt prefix, so the server skips prefill compute for those tokens entirely. - [Why are floating-point formats better than integers for LLM quantization?](https://yourwildcard.ai/blog/why-floating-point-beats-integer-quantization.md): Exponent bits give floating-point formats the dynamic range to keep both tiny values and large outliers in the same tensor. - [Why do GPU kernels break when you upgrade GPUs?](https://yourwildcard.ai/blog/why-gpu-kernels-break-on-upgrades.md): Fast GPU kernels hard-code details of one specific chip, e.g., how much shared memory each block can use. - [Why are GPUs faster than CPUs for AI inference?](https://yourwildcard.ai/blog/why-gpus-beat-cpus-for-inference.md): GPUs are throughput machines. - [Why is image generation compute-bound when the models are small?](https://yourwildcard.ai/blog/why-image-generation-is-compute-bound.md): Image and video models carry roughly 10x fewer parameters than frontier LLMs, but iterative denoising attends over the entire latent object every step, and 50 steps is about 100 forward passes. - [Why don't prefix caching and tensor parallelism help embedding models?](https://yourwildcard.ai/blog/why-llm-optimizations-dont-help-embeddings.md): Embedding models process all tokens in one parallel pass, so there is no decode loop to cache for and nothing worth splitting across GPUs at the 100 MB to 1 GB scale. - [Why do multi-GPU LLM deployments need NVLink or InfiniBand?](https://yourwildcard.ai/blog/why-multi-gpu-needs-nvlink-or-infiniband.md): Multi-GPU inference sits on a bandwidth hierarchy. - [Why does quantization make LLM inference faster in both prefill and decode?](https://yourwildcard.ai/blog/why-quantization-speeds-up-inference.md): Halving precision doubles the effective Tensor Core FLOPS for compute-bound prefill, and it doubles the effective memory bandwidth for memory-bound decode. - [Why should I report P99 latency instead of average?](https://yourwildcard.ai/blog/why-report-p99-latency-not-average.md): LLM response times are right-skewed, so mean latency hides the tail outliers that destroy user trust. - [Why do voice assistants struggle with accents, and whose job is it to fix?](https://yourwildcard.ai/blog/why-voice-assistants-struggle-with-accents.md): Accent failures are dataset coverage failures that surface as product failures, so fixing them is the PM's job and not only the researcher's. - [Will every product manager become an AI product manager?](https://yourwildcard.ai/blog/will-every-pm-become-an-ai-pm.md): Marily Nika's answer is yes. - [Why do worked examples beat case studies in regulated industries?](https://yourwildcard.ai/blog/worked-examples-beat-case-studies-regulated.md): A case study asserts that someone else succeeded.