Check that a PDF citation still points to the same source
Record a quote with its page and file digest, then detect a revised source before reusing the citation.
Record the source version
A filename can stay the same after someone edits the document. Record the source bytes’ SHA-256 digest, the one-based PDF page, and the quoted text. A later check can tell whether you have the same file and whether that page contains the words you cited.
Use the handbook folder and Python environment from the hybrid PDF indexing lab. Save the script alongside pdf_pages.py. It reads the embedded text. For a citation from OCR, keep the reviewed OCR text and record which page image it came from.
View quote_provenance.py
import argparse
from hashlib import sha256
import json
from pathlib import Path
from pypdf.errors import PdfReadError
from pdf_pages import extract_pages
def has_quote(pdf_bytes, page, quote):
pages = extract_pages(pdf_bytes)
return 1 <= page <= len(pages) and quote in " ".join(pages[page - 1][1].split())
def record(pdf, page, quote, target):
pdf_bytes = pdf.read_bytes()
quote = " ".join(quote.split())
if not quote or not has_quote(pdf_bytes, page, quote):
raise ValueError("The quote is missing from the embedded text of that page.")
proof = {
"source": pdf.name,
"sha256": sha256(pdf_bytes).hexdigest(),
"page": page,
"quote": quote,
}
with target.open("x", encoding="utf-8") as file:
json.dump(proof, file, ensure_ascii=False, indent=2)
file.write("\n")
print(f"Recorded page {page} in {target}")
def verify(pdf, target):
proof = json.loads(target.read_text(encoding="utf-8"))
pdf_bytes = pdf.read_bytes()
page = proof["page"]
quote = proof["quote"]
if type(page) is not int or not isinstance(quote, str) or not quote.strip():
raise ValueError("The citation record needs a page number and quote.")
if sha256(pdf_bytes).hexdigest() != proof["sha256"]:
present = "still appears" if has_quote(pdf_bytes, page, quote) else "is missing"
raise ValueError(f"PDF revision changed. The quote {present} on page {page}. Review the citation.")
if not has_quote(pdf_bytes, page, quote):
raise ValueError("The quote is missing from the recorded page.")
print(f"Verified document bytes and page {page}: {quote}")
if __name__ == "__main__":
parser = argparse.ArgumentParser(description="Pin a PDF citation to the source bytes and page.")
commands = parser.add_subparsers(dest="command", required=True)
new = commands.add_parser("record")
new.add_argument("pdf", type=Path)
new.add_argument("page", type=int)
new.add_argument("quote")
new.add_argument("target", type=Path)
check = commands.add_parser("verify")
check.add_argument("pdf", type=Path)
check.add_argument("target", type=Path)
args = parser.parse_args()
try:
if args.command == "record":
record(args.pdf, args.page, args.quote, args.target)
else:
verify(args.pdf, args.target)
except (OSError, KeyError, TypeError, ValueError, PdfReadError) as error:
raise SystemExit(str(error)) from Nonepython quote_provenance.py record pdfs/library-handbook.pdf 1 "Standard book loans last 21 days." citation.json
python quote_provenance.py verify pdfs/library-handbook.pdf citation.json
The citation file contains a digest, page 1, and the quote. The verification command prints the same page and quote. Try recording that quote against page 2. It should reject the mismatched page.
Try a different revision
Open the handbook in a PDF viewer and export page 1 as a separate PDF named page-one.pdf. Point the verifier at the new file and the original record. The quote still appears on page 1, but the file digest changed.
python quote_provenance.py verify page-one.pdf citation.json
Review a changed source before reusing a citation. A policy edit can keep an old sentence while changing its exceptions. A digest establishes file identity. Quote presence and answer relevance are separate checks.
Check your understanding
Why keep the digest when the quote already appears on the right page?
The digest can identify the document version the answer used. The same words may also appear in a different edition.
Sources
Progress is saved in this browser.