Local LLM Field Guide
Search

Intermediate10 min read

Run a model with MLX on Apple silicon

Install MLX LM in a Python environment, generate a reply, and reuse the model from the local cache.

Set up the environment

Use an Apple silicon Mac with a native arm64 Python installation and internet access for the first download. The commands below were checked with Python 3.13. In Terminal, python3 -c "import platform; print(platform.machine())" should show arm64. If it shows x86_64, use a native Terminal and Python.

In a new project folder, create and activate a virtual environment. The version here matches the fine-tuning lab later in the guide.

python3 -m venv .venv
source .venv/bin/activate
python -m pip install "mlx-lm==0.32.2"

Generate your first reply

MLX LM runs models prepared for MLX. For this exercise, use the published 4-bit Qwen weights. The first run downloads the weights and tokenizer into your Hugging Face cache.

mlx_lm.generate \
  --model mlx-community/Qwen2.5-0.5B-Instruct-4bit \
  --prompt "Name one tool used for gardening." \
  --max-tokens 24
You’re ready when

The command prints a gardening tool and generation statistics. Check the output and the memory reported by MLX LM before trying a larger model.

Try it from the cache

HF_HUB_OFFLINE=1 mlx_lm.generate \
  --model mlx-community/Qwen2.5-0.5B-Instruct-4bit \
  --prompt "Complete this sentence with one word: A shovel is a tool for" \
  --max-tokens 16

A reply here confirms that the cached files suffice for this run. If you get an offline cache error, run the first command with connectivity in the same environment and account. Check for a complete model download before retrying offline.

Check your understanding

Can you point this exercise at the GGUF file from the llama.cpp lesson?

Use the MLX weights specified here. GGUF is the packaging format used in the llama.cpp exercise.

Sources

Search the guide