Run a GGUF model with llama.cpp
Choose a llama.cpp build, load an explicit model file, and control context from the command line.
Get the runtime
Install llama.cpp from its official instructions, or choose a build for your operating system and architecture from the release archives. For a first run, a compatible CPU build is sufficient. If you want acceleration, use the release that matches your GPU backend and follow its driver requirements.
The commands below use the standalone llama-cli executable found in release builds. Run it from a terminal with the executable on your PATH. From the extracted binary directory, use ./llama-cli on macOS or Linux, or .\llama-cli.exe in PowerShell.
llama-cli --helpLoad the model file
Download qwen2.5-1.5b-instruct-q4_k_m.gguf. Run this command from the directory containing the file, or supply its full path in quotes.
llama-cli -m "qwen2.5-1.5b-instruct-q4_k_m.gguf" -c 2048-m supplies the model path and -c sets the context in tokens. This GGUF supplies a chat template for the instruction-tuned model. At the prompt, send “Explain a paper calendar in one sentence.” Stop the process with Ctrl+C when you’re done.
The terminal shows a generated sentence and returns to its input prompt. The startup log should identify the model file you supplied.
Try it
Repeat the same prompt at context sizes 2,048 and 4,096. Read the runtime’s startup log for the context and memory allocations. Compare startup times and check whether the larger allocation fits.
Check the file path and download size. Ensure the binary supports the model architecture and GGUF version. If the backend reports a memory allocation error, start with a smaller context or use a smaller model.
Check your understanding
What can you change to test the effect of context while keeping the model the same?
Keep the GGUF path fixed and change the -c value.
Sources
Progress is saved in this browser.