A local-model result needs a record.
“It felt fast” is an observation. A reproducible experiment explains what ran, where, how, and what the result does not establish.
No model experiment was performed for this article. Every field and example value in the downloadable card is blank or explicitly illustrative. No hardware, latency, quality, adoption or cost result is claimed.
Local inference changes several variables at once: the exact model artifact, its quantization, the serving runtime, prompt formatting, sampling settings, cache state and hardware. A result without those details is hard to interpret and harder to repeat.
Record identity before measuring
Question and task
State one bounded question, the intended use, the fixed prompt set and what would count as useful or unsafe.
Model artifact
Record the repository, exact revision or digest, file, base model relationship, quantization, license and model-card limitations.
Runtime and prompt path
Record runtime/version, API endpoint type, chat template or raw mode, context setting, sampling options and stop conditions.
Environment class
Describe CPU/GPU class, accelerator memory, system memory and relevant concurrency—without publishing a machine name, address or private topology.
Runs and evidence
Keep timestamps, warm/cold state, returned counters, outputs, errors and reviewer scores. Preserve failures; do not average them away.
Define the clock you are using
Load time, prompt evaluation and output generation answer different questions. A cold first request can include model loading; a warm request may reuse a resident model or cached prompt tokens. Report them separately where the runtime exposes them.
Ollama’s generate response, for example, documents total duration, load duration, input-token evaluation counts and duration, and output-token counts and duration. Those fields make a more useful record than a stopwatch alone, but they are runtime measurements—not a universal comparison across servers.
- Decide whether streaming is enabled and whether “first token” is measured.
- Separate cold-start and warm-run samples.
- Fix concurrency, prompt order and stopping conditions.
- Report all attempted runs, including errors and timeouts.
- Do not compare token rates when tokenizers, output lengths or APIs differ without explaining the mismatch.
Quality needs a rubric, not a vibe
Use task-specific checks decided before reading outputs. For a structured extraction task, that might mean required fields, unsupported additions, citation fidelity and valid output format. For code, it might mean tests, security constraints and review findings. Keep correctness separate from style and speed.
Correctness
Did the response satisfy known-answer checks and required constraints?
Unsupported claims
Did it invent facts, sources, APIs or certainty not present in the supplied material?
Operational fit
Did the run stay within the declared memory, time, privacy and review boundary?
A single task can show whether a setup deserves another test. It cannot establish that one model is generally “best,” or that another machine will reproduce the same timings.
Use the blank card
The downloadable card is plain text so it can live beside prompts and outputs. Fill it before the first run, then append observed evidence without rewriting the original plan.
Download the blank experiment card (.txt)
What the primary documentation establishes
Model identity and limitations
Hugging Face’s model-card documentation describes model cards as records for intended uses, limitations, training information, datasets, licenses and evaluation results. A card is a starting source; verify the exact artifact being tested.
Runtime counters
Ollama’s generate API reference documents request controls and response counters including loading, prompt evaluation and generation durations. Its API introduction distinguishes local and cloud endpoints, so “Ollama” alone does not prove a run was local.
Serving configuration matters
vLLM’s OpenAI-compatible server documentation is the primary reference for its serving interface and options. Record the exact vLLM version and effective flags rather than assuming API compatibility makes two runtimes equivalent.
Documentation checked. No runtime was installed and no benchmark was executed for this method.