Local LLM Field Guide
Search

Intermediate12 min read

Run on the CPU and check GPU offload

Use one GGUF file to compare CPU inference with GPU offload, then inspect the runtime’s reported placement.

Check available devices

Reuse the Qwen GGUF and llama.cpp setup from the llama.cpp setup lesson. Start with a release built for your operating system and chip. For GPU acceleration, choose a build that supports your GPU and check the runtime’s driver instructions. Metal is one path on Apple silicon. CUDA is one path on supported NVIDIA systems. Supported AMD paths vary by operating system and build.

llama-cli --list-devices

On Apple silicon, a Metal build should list a Metal device. On a discrete GPU setup, look for your card under the selected backend. If your build lists no accelerator, continue with the CPU example.

Run a CPU baseline

From the folder containing the GGUF file, run a short single-turn prompt. Use ./llama-cli or .\llama-cli.exe if the executable is in the current folder and outside your PATH.

llama-cli -m "qwen2.5-1.5b-instruct-q4_k_m.gguf" -c 2048 --device none -ngl 0 --verbose -st -n 64 --temp 0 -p "In four words or fewer, what is one use for a paper calendar?"

--device none disables accelerator offload and -ngl 0 requests zero model layers on the GPU. --verbose exposes the layer placement log. Look for a line reporting zero layers offloaded, even if the runtime lists a GPU at startup.

Try GPU offload

If a supported GPU is available, repeat the same request with -ngl 99. That value requests more layers than this small model has, so the runtime can offload the available layers if they fit.

llama-cli -m "qwen2.5-1.5b-instruct-q4_k_m.gguf" -c 2048 -ngl 99 --verbose -st -n 64 --temp 0 -p "In four words or fewer, what is one use for a paper calendar?"
You’re ready when

You can record the actual layer placement for each run. In one test with build b12368 on Apple silicon, the log reported 0 of 29 layers offloaded for the CPU run and 29 of 29 for Metal. If loading fails, try a lower layer count such as -ngl 4, a smaller context, or a smaller model. For Ollama, use ollama ps during a loaded run to see processor placement.

Check your understanding

The startup log lists your GPU on a run with -ngl 0. Has it loaded the model layers onto the GPU?

Device discovery only shows the device is available. Inspect the layer placement or memory breakdown for the actual run.

Sources

Search the guide