Weights On Disk

Updated 2026-07-07

Quantization means storing a model's weights with fewer bits per weight than the precision they were trained in, so the file on disk gets smaller. A "1-bit model" takes this to the extreme and stores each weight as a single sign bit plus a shared scale for its group, so a weight that would take 16 bits in FP16 takes about 1.125 bits on disk. On this page you learn how those bits are laid out in a block and how to decode one block back into weights with a few lines of Python.

DataArchitectureTrainingWeights on disk1.15 GB GGUFRuntimeprefill, KV cache, decodeKernelsMetal / CUDATokens outtok/s and energyEvaluationevaluation feeds the next round of data and training
The PrismML system at every layer. The highlighted stage is the part this guide explains.

GGUF is the file format that llama.cpp and related runtimes use to store a model as one file, holding the quantized weights plus the metadata a runtime needs to load them. The Bonsai files you download from Hugging Face, e.g., Bonsai-8B-Q1_0.gguf, are GGUF files.

A ternary model is a close relative of a 1-bit model. Each weight takes one of three values, -1, 0, or +1, times the group scale, instead of two. The extra zero value costs a little compression and buys back some quality.

This guide pairs with Lab 05 of Inference The Hard Way, which has you parse one weight tensor by hand. Once you can do that, you can open a compressed model file and confirm byte by byte what it holds.

Read the Q1 block layout

The course describes a group of 128 weights stored as one FP16 scale plus 128 packed sign bits. To reconstruct a weight, you take its sign bit b and compute scale times (2b - 1), which gives plus or minus the scale.

One FP16 weight costs 16 bits. The Q1_0_g128 format costs one sign bit per weight plus one FP16 scale shared by 128 weights: 1 + 16/128 = 1.125 bits per weight. That is how 16.38 GB becomes 1.15 GB.
group:
  scale: FP16
  bits: 128 sign bits

reconstruction:
  w = scale x (2b - 1)

The storage cost per weight follows from the block:

(2 bytes x 8 bits/byte + 128 bits) / 128 weights = 1.125 bits/weight

So "1-bit" is honest about the sign bits but rounds away the scale. The scale costs 16 bits per 128 weights, which adds the extra 0.125 bits.

Decode a block in Python

This sketch reads one block from an open file handle. It reads the 2-byte scale, unpacks the 16 bytes of sign bits, and reconstructs the 128 weights.

import numpy as np

scale = np.frombuffer(f.read(2), dtype=np.float16)[0]
bits = np.unpackbits(np.frombuffer(f.read(16), dtype=np.uint8))
weights = scale * (2.0 * bits - 1.0)

This sketch comes from the course material, and we have not yet run it against a real model file. Before you trust it on a real GGUF file, check the scale order, the bit order, the tensor offset, and the block struct against the runtime implementation and the GGUF metadata.

Know what the file format does not give you

A smaller file is not the whole story. The savings only help at inference time if the runtime keeps the weights packed during execution. The kernel must unpack the sign bits and multiply by the scale on the fly, without expanding the tensor back into a full FP16 copy in memory. If it expands the tensor first, you paid for compression on disk and got none of it at run time.

Next steps

See something wrong? Fix it.