How to serve Ternary Bonsai 27B through an OpenAI compatible API

Start mlx_lm.server with the PrismML model ID, bind it to localhost, and point your client at http://127.0.0.1:8080/v1. The endpoint supports the common chat completions shape used by local agents and application libraries.

Start the server

mlx_lm.server \
  --model prism-ml/Ternary-Bonsai-27B-mlx-2bit \
  --host 127.0.0.1 \
  --port 8080

Check the model endpoint

A successful response should list prism-ml/Ternary-Bonsai-27B-mlx-2bit. Send a short chat completion only after the model download and load have finished.

Keep the server on 127.0.0.1 unless another device must connect. A server bound to every network interface needs access controls and firewall rules.

curl http://127.0.0.1:8080/v1/models

Questions people ask

Do I need an API key for a local MLX server?

A local server usually does not verify an API key. Some client libraries still require a nonempty placeholder value.

Can I use the same endpoint with an agent?

Yes, if the agent accepts a custom OpenAI compatible base URL and supports the model's response format.

Sources

The benchmark numbers on this page describe one checkpoint, runtime, machine, and test shape. Reproduce the test on the hardware and workload you plan to use before making a product decision.