Local LLM Field Guide
Search

Intermediate12 min read

Stream a local model’s reply

Print model text as it arrives, measure time to first visible text, and detect an interrupted stream.

Read one event at a time

The earlier Python script asks Ollama to wait and send one JSON response. With "stream": True, the chat endpoint sends newline-delimited JSON. Read one line at a time, parse it, and print any message.content text as it arrives. Some events can carry only status or other fields.

Run the client

Keep Ollama running with the quickstart model available. Save the script below in your project folder. Use python3 on macOS or Linux, or py -3 on Windows if python is unavailable.

Download stream_chat.py
View stream_chat.py
import argparse
import json
import sys
from time import perf_counter
from urllib.error import HTTPError, URLError
from urllib.request import Request, urlopen

parser = argparse.ArgumentParser(description="Stream a reply from local Ollama.")
parser.add_argument("prompt")
args = parser.parse_args()

payload = {
    "model": "qwen2.5:1.5b",
    "messages": [{"role": "user", "content": args.prompt}],
    "options": {"num_predict": 160},
    "stream": True,
}
request = Request(
    "http://127.0.0.1:11434/api/chat",
    data=json.dumps(payload).encode("utf-8"),
    headers={"Content-Type": "application/json"},
    method="POST",
)

started = perf_counter()
first_text_at = None
finished = False
try:
    with urlopen(request, timeout=120) as response:
        for line in response:
            if not line.strip():
                continue
            event = json.loads(line)
            if event.get("error"):
                raise SystemExit(f"\nOllama stopped the stream: {event['error']}")
            text = event.get("message", {}).get("content", "")
            if text:
                if first_text_at is None:
                    first_text_at = perf_counter() - started
                print(text, end="", flush=True)
            if event.get("done"):
                finished = True
                break
except HTTPError as error:
    raise SystemExit(f"Ollama returned HTTP {error.code}. Check the model tag.") from None
except (URLError, OSError, ValueError) as error:
    raise SystemExit(f"\nCould not finish the stream: {error}") from None

print()
if not finished:
    raise SystemExit("The connection closed before Ollama reported completion.")
if first_text_at is None:
    raise SystemExit("Ollama finished without emitting visible text.")
print(f"First text in {first_text_at:.2f} s.", file=sys.stderr)
python stream_chat.py "Write three short tips for organizing a paper notebook."
You’re ready when

The reply appears in pieces and the terminal prints “First text in … s.” That time measures from the start of the client request to the first visible text, including startup if the model was cold. A fast model can make the pieces look almost simultaneous.

Compare a cold run and a warm run

Run ollama stop qwen2.5:1.5b if the model is loaded. Run the same client command twice. Record the first-text time in both cases. The first run can include model loading. Repeat with a longer prompt to see how input processing affects the wait.

In a chat interface, show partial text while the request runs. Check the final done event before saving the reply as complete. If the connection ends early or Ollama sends an error event, show that the partial output was interrupted.

Check your understanding

Why does the script wait for nonempty message.content before recording the first-text time?

The first JSON line may carry metadata. The first nonempty text is when the reader can begin to see the answer.

Sources

Search the guide