Local LLM Field Guide
Search

Intermediate16 min read

Read queue and generation metrics from a local server

Sample a protected llama.cpp metrics endpoint during a repeatable workload and distinguish gauges, counters, and client latency.

On this page

Take an idle sample

Use the authenticated loopback server from the server authentication lab. It started with --metrics. Save this script next to compatible_chat.py, set LOCAL_MODEL_API_KEY in the client shell, and take one sample.

Download sample_server_metrics.py
View sample_server_metrics.py
import argparse
import os
from time import sleep
from urllib.error import HTTPError
from urllib.request import ProxyHandler, Request, build_opener

from compatible_chat import NoRedirects, local_base

METRICS = {
    "llamacpp:requests_processing": "active",
    "llamacpp:requests_deferred": "waiting",
    "llamacpp:tokens_predicted_total": "generated_total",
}


def read_metrics(base):
    key = os.environ.get("LOCAL_MODEL_API_KEY")
    headers = {"Authorization": f"Bearer {key}"} if key else {}
    endpoint = base.removesuffix("/v1") + "/metrics"
    opener = build_opener(ProxyHandler({}), NoRedirects())
    with opener.open(Request(endpoint, headers=headers), timeout=5) as response:
        body = response.read().decode("utf-8")
    values = {}
    for line in body.splitlines():
        parts = line.split()
        if len(parts) >= 2 and parts[0] in METRICS:
            values[METRICS[parts[0]]] = float(parts[1])
    if len(values) != len(METRICS):
        raise ValueError("Expected llama.cpp metrics are missing. Check your server build.")
    return values


if __name__ == "__main__":
    parser = argparse.ArgumentParser(description="Sample metrics from your own local llama.cpp server.")
    parser.add_argument("--base", type=local_base, default="http://127.0.0.1:8080/v1")
    parser.add_argument("--samples", type=int, choices=range(1, 31), default=12)
    args = parser.parse_args()
    try:
        for index in range(args.samples):
            values = read_metrics(args.base)
            print(f"{index + 1:02d} | active={values['active']:.0f} | waiting={values['waiting']:.0f} "
                  f"| generated_total={values['generated_total']:.0f}", flush=True)
            if index + 1 < args.samples:
                sleep(1)
    except HTTPError as error:
        raise SystemExit(f"Metrics returned HTTP {error.code}. Check --metrics and the local API key.") from None
    except (OSError, ValueError) as error:
        raise SystemExit(str(error)) from None
python sample_server_metrics.py --samples 1

The script reads active requests, deferred requests, and total generated tokens. Active and waiting counts are gauges. They describe the moment you sampled and can go up or down. The generated-token total is a counter for that server run.

Sample during a workload

In the first client terminal, start collecting samples. In another client terminal with the same key, run the workload from the concurrent requests lab.

python sample_server_metrics.py --samples 20
python measure_concurrency.py --base http://127.0.0.1:8080/v1 --model qwen-local --count 8
You’re ready when

The generated-token total grows between the first and last samples. Active or waiting counts may stay at zero if requests finish between samples. If you need to see waiting work, try two clients with one server slot and repeat.

Read the counters and keep a baseline

Subtract the first counter value from the last for the tokens generated over that interval. Divide by the elapsed seconds for an average rate. A restart can reset the counter, so start a new measurement interval after restarting.

Record the server build, model, context per slot, client count, and output cap. Compare gauges with the client’s measured median reply time. A readiness check or a single gauge cannot tell you what a client waited for.

Check your understanding

Why might every waiting sample be zero while some requests still waited?

A queue can form and drain between observations. Use client timings and a repeatable workload to investigate.

Sources

Search the guide