Local LLM Field Guide
Search

Beginner8 min read

Hardware and quantization

Estimate the space weights need, leave room for context, and pick a sensible model size for your machine.

Read your machine

Write down free disk space and total system memory. If you have a dedicated graphics card, find its dedicated GPU memory (VRAM). On Apple silicon the CPU and GPU share unified memory with the operating system and other apps. On a system with separate memory pools, how much of the model can sit in VRAM affects speed. Supported acceleration depends on the runtime, hardware, and drivers.

Estimate weight size

Quantization represents weights with fewer bits. A rough estimate for raw weight storage is parameters × bits per parameter ÷ 8. With parameters in billions, the result is decimal gigabytes. Real files can be larger because of metadata, scales, and tensors kept at different precisions. Runtime memory also includes the context cache, work buffers, and the app.

Raw weights3.5 GB

Try 7 billion parameters at 4 bits and then at 16 bits. The rough weight estimate changes from 3.5 GB to 14 GB. A label such as Q4_K_M names a specific GGUF quantization, so compare the listed download size too. Lower precision can hurt output quality. Speed gains depend on the runtime and hardware.

Start small and measure

  1. Choose a small instruction model with a low-precision variant, such as the 1.5B example in the setup lessons.
  2. Run a short prompt while checking system memory or VRAM. On Ollama, ollama ps shows how the loaded model is placed.
  3. Increase model size or context one setting at a time and try the same prompt again. Record the change in speed and response quality.
If the model stalls or fails to load

Close other memory-heavy apps, reduce the context, or choose a smaller model or quantization. Use the runtime’s loading message and system memory readout to identify which change helped.

Check your understanding

Will a model whose file is 7.8 GB fit fully in 8 GB of dedicated GPU memory?

The file size alone cannot tell you. The runtime needs extra memory, and the file includes data with its own layout and overhead. Try a smaller variant and measure how much actually loads onto the GPU.

Sources

Search the guide