Measure local inference performance

Measure different parts of the wait

Load time

Time to make model weights available. A cold run includes more loading work than a warm run.

Time to first token

Time from sending the prompt to the first generated token. Measure it with a streaming client and exclude any initial chunks that carry metadata alone.

Decode speed

How quickly the model generates output once it has begun. Longer prompts and larger contexts can affect the result.

Collect timings

Save this as measure_chat.py alongside the local API script. Ollama reports durations in nanoseconds. The response uses eval_count for generated tokens and eval_duration for their evaluation time.

from local_chat import chat

result = chat({
    "model": "qwen2.5:1.5b",
    "messages": [{"role": "user", "content": "Describe five library services."}],
    "options": {"temperature": 0, "num_predict": 128, "num_ctx": 2048},
    "stream": False,
})
print(f"Load: {result['load_duration'] / 1e9:.2f} s")
print(f"Total: {result['total_duration'] / 1e9:.2f} s")

tokens = result.get("eval_count", 0)
seconds = result.get("eval_duration", 0) / 1e9
if seconds:
    print(f"Decode: {tokens / seconds:.1f} tokens/s")

With stream: false, the client receives the complete response at once. The reported timing fields can describe server work, but this script cannot time the arrival of the first token.

Run a fair comparison

1. Choose a fixed prompt and output cap. Keep temperature, context length, hardware, and other workloads steady.
2. Run ollama stop qwen2.5:1.5b if the model is loaded. Run the script once, then run it three more times while the model is warm. Record cold load time, median warm decode speed, and memory or GPU placement.
3. Change one setting, such as quantization, and repeat. Check that answer quality still meets your task’s requirements.

Check your understanding

One run is fast because the model was already loaded. Which measurements can you compare across two warm runs?

Compare warm timings under matching conditions. Record cold load separately.

Sources

• Ollama · Chat timing fields
• Ollama · Context length and offload
• llama.cpp · llama-bench