Local LLM Field Guide
Search

Intermediate16 min read

Rerank retrieved passages with a local model

Score question–passage pairs, then measure whether reranking improves the first result on the sample handbook.

On this page

Score a short candidate list

The PDF lab retrieves a handful of passages. A cross-encoder reads each question and candidate together and produces a score for that pair. Use it after retrieval so it evaluates only the candidates. It can reorder the list, and its score is not a guarantee that the passage answers the question.

Use the folder and database from the handbook citation lab, with the original PDF indexed and page 4 skipped. Keep check_handbook.py from that lesson and download the script below. Use the same Python environment and add the tested versions. The first run downloads the reranker.

Download rerank_handbook.py
View rerank_handbook.py
import argparse
from pathlib import Path
import sqlite3

from sentence_transformers import CrossEncoder

from check_handbook import CASES
from handbook_search import DEFAULT_DB, search

MODEL = "cross-encoder/ms-marco-MiniLM-L6-v2"
REVISION = "233902d25c440f23af6f7d6e94d2946bac0bee0a"


def rerank(model, question, candidates):
    if not candidates:
        return []
    pairs = [(question, passage.text) for passage in candidates]
    scores = model.predict(pairs, show_progress_bar=False)
    return sorted(zip(candidates, scores), key=lambda item: float(item[1]), reverse=True)


def print_results(model, database, question, mode, count):
    candidates = search(database, question, mode=mode, limit=count)
    ranked = rerank(model, question, candidates)
    print("Candidate order:", ", ".join(f"{hit.source} p.{hit.page}" for hit in candidates) or "none")
    print("Reranked:")
    for hit, score in ranked:
        print(f"  {score:.2f} | {hit.source}, page {hit.page}")


def evaluate(model, database, mode, count):
    retrieved = first_before = first_after = 0
    for question, expected in CASES:
        candidates = search(database, question, mode=mode, limit=count)
        reranked = rerank(model, question, candidates)
        before = [p.page if p.source == "library-handbook.pdf" else None for p in candidates]
        after = [p.page if p.source == "library-handbook.pdf" else None for p, _ in reranked]
        retrieved += expected in before
        first_before += bool(before and before[0] == expected)
        first_after += bool(after and after[0] == expected)
        print(f"Page {expected} | before {before} | after {after} | {question}")
    print(f"Candidate coverage: {retrieved}/{len(CASES)}")
    print(f"First result: {first_before}/{len(CASES)} before, {first_after}/{len(CASES)} after")


if __name__ == "__main__":
    parser = argparse.ArgumentParser(description="Rerank retrieved PDF passages on the CPU.")
    parser.add_argument("--db", type=Path, default=DEFAULT_DB)
    parser.add_argument("--mode", choices=("keyword", "vector", "hybrid"), default="hybrid")
    parser.add_argument("--candidates", type=int, choices=range(1, 9), default=3)
    parser.add_argument("--offline", action="store_true")
    parser.add_argument("question", nargs="?")
    args = parser.parse_args()
    try:
        model = CrossEncoder(MODEL, revision=REVISION, device="cpu", max_length=256,
                             local_files_only=args.offline, model_kwargs={"use_safetensors": True})
        if args.question:
            print_results(model, args.db, args.question, args.mode, args.candidates)
        else:
            evaluate(model, args.db, args.mode, args.candidates)
    except (OSError, RuntimeError, sqlite3.Error, ValueError) as error:
        raise SystemExit(str(error)) from None
python -m pip install "sentence-transformers==5.6.1" "transformers==4.57.6"
python rerank_handbook.py --mode keyword "Which station can digitize a paper photograph?"
You’re ready when

The original keyword ranking puts page 1 ahead of page 2 for this question. The reranker can move page 2, which contains the scanner, to the top. Check the page rather than relying on the score alone.

Measure coverage and ranking

python rerank_handbook.py --offline --mode keyword
python rerank_handbook.py --offline --mode hybrid
python rerank_handbook.py --offline --mode keyword --candidates 1

With no question supplied, the script scores the eight handbook queries. In one run, keyword search went from 6 of 8 to 8 of 8 first results with three candidates. Hybrid search already had 8 of 8 first results on that set. With one candidate, there is nothing to reorder.

Try it on your own questions

Add questions before choosing a candidate count. Record how often the candidate list contains the right page, how often it appears first, and how much time reranking adds. If the right page is missing, investigate extraction or retrieval first.

Check your understanding

Can reranking find a page that the retriever left out?

It only scores the passages it receives. Improve candidate coverage if the needed page is missing.

Sources

Search the guide