gnostral.rs

The reference machine is not a datacenter server: an RTX 3070 with 8 GB of graphics memory and 32 GB of main memory. It is a personal-workstation configuration — and that is exactly what makes the research interesting.

The headline result: by keeping the always-used parts on the card and borrowing the quantised experts from main memory, throughput goes from 9.51 to 20.18 tokens per second, or 2.12 times faster. Measured locally, three repetitions per condition, responses identical byte for byte.

A "2.12×" is a median. Behind it are twelve measured answers: three processes without the patch and three with it, each asked for a 128-token then a 512-token answer. The two clouds never touch: the slowest patched answer (19.48 tokens/s) is still twice as fast as the fastest unpatched one (10.12).

Milestone Q-012 asks another question: can the same executable serve models of different kinds? Three families, one after another: a dense model that writes text, an embeddings model that turns sentences into vectors, and the large quantised MoE. All three answer correctly — the embeddings model, for instance, rates two related sentences much closer (0.78) than two unrelated ones (0.17).

Above all, the chart shows where each model puts its weights. The 30-billion-parameter MoE takes 2.4 GB on the card and 19 GB in main memory: that is the dossier's trick, measured. The first run, built without the expert-residency feature, climbed to 6.8 GB on the card and timed out; it stays on record, and a guard now refuses a binary that forgets it.

A "2.12× faster" means nothing without its point of comparison. Here, the reference is not another piece of software: it is the same engine, one version earlier.

Concretely, "Strata" is the revision of mistral.rs used in the lab: public version 0.9.3, plus an expert-residency integration. The 9.51 tokens/s measure that engine before the patch; the 20.18 measure the same binary, rebuilt with a single function changed — the quantised expert weights are borrowed instead of being copied on every call.

Against llama.cpp, on the other hand, the comparison exists and it is unfavourable: 34.54 tokens/s for llama.cpp against 20.18 for gnostral, on the same model and the same card. The repository's report says so explicitly: these gains do not establish superiority over llama.cpp's CPU-MoE options.

As for mistral.rs, it is not an opponent: it is the engine in which the patch is applied. gnostral is not a competing engine — it is a set of patches, typed contracts and measurement tools around mistral.rs. Comparing against it would be like comparing a car with its own engine.

The project's most solid contribution may not be the speed gain, but the method. Six steps, always the same, and none can be skipped.

Concretely: the code and the model are identified by a fingerprint (SHA), a plan describes exactly what will run, a single process is started within limits, a precise answer is checked, the process is stopped, the memory is checked to have come back, and then an independent program redoes the checks.

Order matters. A server can answer a health check while its weights are not yet loaded. gnostral.rs therefore distinguishes eight states, from the mere fact of being "declared in the documentation" up to "qualified by an independent audit".

The loop's sixth step — stopping and recovery — is the one that fails the most runs. After each shutdown, what is still occupied on the graphics card is measured. The gate is set before the run: ±128 MiB. A fast run that leaves memory behind is refused.

The first Q-010 run failed: graphics memory did not come back within the expected margin (+196 MiB against a ±128 MiB tolerance). The first Q-012 run failed too: a compile-time feature was missing, and the MoE model timed out.

Both failures are still on the record. It is a methodological choice, not an affectation: a result whose misses you do not know is not reusable. And a figure that cannot be contradicted is not a measurement; it is a claim.

The same principle applies to the Bonsai benchmark: the reference kept is 35.2 tokens per second, not the more flattering claim of 40 — for lack of sufficient evidence.

gnostral.rs moves forward by quests: each opens a precise question and closes with a written decision, including when the decision is "not qualified". The public log counts fifteen between October 7 and 21, 2026.

Part of the work is examining other engines before touching them. The repository keeps its refusals as carefully as its adoptions:

The very first quest, Q-001, measured nothing new: it withdrew a figure. A Bonsai benchmark claimed 40 tokens per second; for lack of sufficient evidence, the repository kept 35.2 and never published the 40. So the project starts with a correction — a choice of method.

The target fits in one sentence: an inspectable, Rust-first, local inference control plane for constrained hardware — choosing the right engine for each model, bounding resources, proving readiness, stopping cleanly, accounting for performance. Compute kernels and expert placement stay the business of each engine.