Measure local inference performance
Compare startup time, generation speed, and memory under the same workload.
Learning path
Go deeper
Measure quality and speed, train an adapter or a classifier, and check how a local service behaves under load.
Performance and serving
4 lessonsMeasure what happens when requests overlap
Compare one and two simultaneous clients, then separate throughput, response time, and memory use.
Read queue and generation metrics from a local server
Sample a protected llama.cpp metrics endpoint during a repeatable workload and distinguish gauges, counters, and client latency.
Measure the speed and recall of a vector index
Compare exact search with HNSW on 30,000 local vectors and measure how search effort changes recall, latency, and index size.
Evaluation and reliability
7 lessonsEvaluate models on your own tasks
Build a small test set, score outputs against a rubric, and compare revisions fairly.
Test a local app before updating its model
Run a repeatable check of response shape and task accuracy, record the model identity, and compare a candidate with your baseline.
Compare a reasoning model’s thinking settings
Run the same checked problem twice, then compare answer accuracy, token use, and response time.
Check that a PDF citation still points to the same source
Record a quote with its page and file digest, then detect a revised source before reusing the citation.
Evaluate local speech recognition in two languages
Compare English and Spanish transcripts against reference words and score whether important details survived.
Evaluate retrieval and citations across document types
Score six labeled questions with different evidence types and compare candidate coverage, answer relevance, and missing-answer behavior.
Choose which source governs a conflicting answer
Test a case where an older handbook and a dated policy mention different values, then make the relevant source rule explicit.
Training and adaptation
4 lessonsDecide when to fine-tune
Compare prompting, retrieval, and adapters before committing time and compute to training.
Train and evaluate a LoRA adapter
Train an adapter on fictional library requests with MLX LM, then score it against the same base model on held-out examples.
Train a small classifier and compare it with a chat model
Classify fictional library requests with scikit-learn, check when to send a case for review, and compare a local LLM baseline.
Train a tiny causal Transformer on the CPU
Train a single attention block on fictional sentences and compare training loss, held-out predictions, and the resulting completion.
Tool execution
1 lessonGive a local model a read-only tool
Learn the request–execute–reply loop with a local dictionary and explicit argument validation.