Intermediate15 min read

Use one local client with several runtimes

Run Chat Completions against Ollama and llama.cpp, then use the same client with LM Studio’s local server.

On this page

Agree on the request

The earlier Python example uses Ollama’s native /api/chat. Several local runtimes also implement /v1/chat/completions. A shared client can use that request and read the reply at choices[0].message.content. Check each runtime’s documentation before using tools, streaming, images, or structured output through that interface.

Use Python 3 and save the client below. It accepts only local loopback URLs and reads a local server key from LOCAL_MODEL_API_KEY when you set one. For the unauthenticated loopback examples, leave that variable unset.

Download compatible_chat.py

View compatible_chat.py

import argparse
import json
import os
from urllib.error import HTTPError
from urllib.parse import urlsplit
from urllib.request import HTTPRedirectHandler, ProxyHandler, Request, build_opener


class NoRedirects(HTTPRedirectHandler):
    def redirect_request(self, req, fp, code, msg, headers, newurl):
        return None


def local_base(value):
    url = urlsplit(value)
    if (
        url.scheme != "http"
        or url.hostname not in {"localhost", "127.0.0.1", "::1"}
        or url.username is not None
        or url.password is not None
        or url.path.rstrip("/") != "/v1"
        or url.query
        or url.fragment
    ):
        raise argparse.ArgumentTypeError("Use a loopback HTTP address ending in /v1.")
    return value.rstrip("/")


def request(base, endpoint, payload=None, *, timeout=120):
    headers = {"Content-Type": "application/json"}
    key = os.environ.get("LOCAL_MODEL_API_KEY")
    if key:
        headers["Authorization"] = f"Bearer {key}"
    data = json.dumps(payload).encode("utf-8") if payload is not None else None
    message = Request(base + endpoint, data=data, headers=headers)
    opener = build_opener(ProxyHandler({}), NoRedirects())
    with opener.open(message, timeout=timeout) as response:
        return json.load(response)


def main():
    parser = argparse.ArgumentParser(description="Talk to a local Chat Completions endpoint.")
    parser.add_argument("--base", type=local_base, default="http://127.0.0.1:11434/v1")
    parser.add_argument("--list", action="store_true", help="Print available model IDs")
    parser.add_argument("--model", help="Exact model ID from --list")
    parser.add_argument("prompt", nargs="?")
    args = parser.parse_args()
    if not args.list and (not args.model or not args.prompt):
        parser.error("Use --list, or provide --model and a prompt.")

    try:
        if args.list:
            for model in request(args.base, "/models")["data"]:
                print(model["id"])
            return

        result = request(args.base, "/chat/completions", {
            "model": args.model,
            "messages": [{"role": "user", "content": args.prompt}],
            "temperature": 0,
            "max_tokens": 96,
            "stream": False,
        })
        text = result["choices"][0]["message"].get("content")
        if not isinstance(text, str) or not text.strip():
            raise ValueError("The server returned no visible reply. Check the model and output limit.")
        print(text)
    except HTTPError as error:
        raise SystemExit(f"Server returned HTTP {error.code}. Check the model ID, port, and API key.") from None
    except (OSError, KeyError, IndexError, TypeError, ValueError) as error:
        raise SystemExit(f"Could not read a local reply: {error}") from None


if __name__ == "__main__":
    main()

Try Ollama

Keep Ollama running with the quickstart model installed. List the models and choose an ID from the result.

python compatible_chat.py --list
python compatible_chat.py --model qwen2.5:1.5b "Name one service a public library provides."

Switch to llama.cpp

In a separate terminal, use the GGUF file from the llama.cpp setup guide. From the model’s folder, start the server on loopback. If llama-server is outside your PATH, use its full path or run it from the release directory with an absolute model path.

llama-server --host 127.0.0.1 --port 8080 --alias qwen-local -m "qwen2.5-1.5b-instruct-q4_k_m.gguf" -c 4096

In the client terminal, point to the new address and use the listed alias.

python compatible_chat.py --base http://127.0.0.1:8080/v1 --list
python compatible_chat.py --base http://127.0.0.1:8080/v1 --model qwen-local "Name one service a public library provides."

Try LM Studio

In LM Studio, open the Developer tab and start the local server with a downloaded model available. Leave local network serving disabled for this lab. Use the URL shown by LM Studio, usually http://127.0.0.1:1234/v1, and run the client with --list. Use one of the returned chat model IDs with --model.

You’re ready when

The same script lists model IDs and prints replies from two runtimes. If a request fails, check the address, model ID, and whether your local server requires a token. Stop any server you started for the exercise when you finish.

Check your understanding

Does a model ID returned by Ollama automatically identify the same model on llama.cpp or LM Studio?

List the models on the server you’re calling. Each runtime can use its own IDs or aliases.