Ask questions about local documents

Retrieve evidence first

Retrieval-augmented generation (RAG) finds relevant passages and puts them alongside your question in the model’s context. An embedding model represents passages and questions as vectors. You can compare their directions with cosine similarity to rank likely matches.

Keep Ollama running and the chat model from the quickstart downloaded. Pull the small embedding model once.

ollama pull all-minilm

Build a tiny local search

Save this as ask_notes.py. It embeds three fictional library notes and the question, prints the two best matches, and asks the chat model to quote a relevant sentence. The truncate flag makes overlong input fail visibly.

import json
import math
from urllib.request import Request, urlopen

NOTES = [
    ("S1", "The library has a 3D printer. Reserve it at the media desk."),
    ("S2", "Standard book loans last 21 days. Renewals depend on holds."),
    ("S3", "The reading room closes at 6 p.m. on Saturdays."),
]

def post(endpoint, payload):
    request = Request(
        "http://127.0.0.1:11434" + endpoint,
        data=json.dumps(payload).encode("utf-8"),
        headers={"Content-Type": "application/json"},
        method="POST",
    )
    with urlopen(request, timeout=180) as response:
        return json.load(response)

def cosine(left, right):
    scale = math.sqrt(sum(x * x for x in left) * sum(y * y for y in right))
    return sum(x * y for x, y in zip(left, right)) / scale if scale else 0

question = "When does the reading room close on Saturdays?"
texts = [text for _, text in NOTES]
vectors = post("/api/embed", {
    "model": "all-minilm",
    "input": texts + [question],
    "truncate": False,
})["embeddings"]

ranked = sorted(
    zip(NOTES, vectors[:-1]),
    key=lambda row: cosine(row[1], vectors[-1]),
    reverse=True,
)[:2]
passages = {label: text for (label, text), _ in ranked}
evidence = "\n".join(f"[{label}] {text}" for label, text in passages.items())
print(evidence, "\n")

reply = post("/api/chat", {
    "model": "qwen2.5:1.5b",
    "messages": [
        {"role": "system", "content": (
            "Copy one sentence exactly from the passages that answers the question. "
            "Return it as quote and its passage ID as source, without brackets. "
            "Match the exact day, place, and whether the question asks about opening or closing. "
            "If no passage answers the question, use null for both fields."
        )},
        {"role": "user", "content": f"Passages:\n{evidence}\n\nQuestion: {question}"},
    ],
    "format": {
        "type": "object",
        "properties": {
            "quote": {"type": ["string", "null"]},
            "source": {"enum": [*passages, None]},
        },
        "required": ["quote", "source"],
        "additionalProperties": False,
    },
    "options": {"temperature": 0},
    "stream": False,
})
try:
    answer = json.loads(reply["message"]["content"])
    quote, source = answer["quote"], answer["source"]
except (json.JSONDecodeError, KeyError, TypeError):
    raise SystemExit("The model did not return a usable answer.") from None

if isinstance(source, str) and isinstance(quote, str):
    if quote.strip() and quote in passages.get(source, ""):
        print(f"[{source}] {quote}")
        raise SystemExit(0)

print("No verified quote found.")

Test an answer and an unknown

For the first question, look for [S3] The reading room closes at 6 p.m. on Saturdays. in the answer after the retrieved passages. Try “What is the name of the library founder?” The notes lack that fact, so the desired result is “No verified quote found.”

Then ask “What time does the reading room open on Mondays?” A genuine quote about Saturday closing can pass the quote check and still fail to answer the question. Compare the place, day, and action yourself. The top two passages always come back, even when neither answers the question.

Check your understanding

Do you need to retrain the model when the library changes its opening hours?

Update the passage, re-embed it, and test retrieval and answers again. The chat model can use the new passage in context.

Sources

• Ollama · Embeddings endpoint
• Ollama Library · all-minilm
• Lewis et al. · Retrieval-Augmented Generation