HybridLink on Two Pascal GPUs: Tokens per Second Was the Wrong Metric

I spent a day running Together's HybridAgent checkpoints on two 2016 TITAN X Pascal cards, trying to find out whether latent reasoning gets a local model to a correct answer sooner. My first metric was tokens per second. It could not answer the question, and my first set of numbers came from a model with the interesting part switched off.

The measurement that could answer it was a same-checkpoint on/off comparison with an external grader. On six fixed arithmetic questions, three seeds each, enabling HybridLink gave the 8B model a median 2.01× speedup across the eight pairs where both modes finished correctly, and raised correct completions from 11/18 to 13/18. The 14B model's median paired speedup was 1.00×, but it went from 7/18 correct to 12/18. Time to first reasoning text did not move at either size. Visible tokens per second went down in both cases.

Those three numbers answer different questions, and conflating them is how a latent-reasoning result gets misreported in either direction.

What HybridLink does in the released code

The paper's idea is that a reasoning model should not have to put every intermediate thought through the vocabulary. Inside a <latent> span, the engine takes the transformer's last hidden state, passes it through a small adapter the authors call HybridLink, and feeds the result back as the next input embedding. No token is sampled; the KV context carries on. The released checkpoints are Qwen3-8B-Base and Qwen3-14B-Base after the authors' hybrid supervised and reinforcement training, shipped as a backbone plus a separate adapter.pt.

Two things in the released engine matter for measurement. A latent step still runs the full transformer and still produces vocabulary logits, because the exit decision reads them. So a latent step costs real compute and a KV-cache slot while emitting nothing readable. And the loop lives in the authors' Python engine, not in the weights: load the backbone through any runtime that does not implement it and you have a plain Qwen3 derivative.

The paper reports 1.8–2.6× faster inference and 35.5–54.7% fewer tokens on H100s in an agentic setup. Those are the authors' numbers; nothing below reproduces them.

Wrong turn one: Ollama timed the backbone

The quick route was community GGUF conversions through Ollama. They ran. Warm short-prompt decode was about 53.0 tok/s for the 8B and 23.6 tok/s for the 14B at Q4_K_M through Vulkan, with Qwen3 instruct tags alongside as a rough reference. Pleasant numbers for ten-year-old silicon.

They measure nothing about HybridLink. The adapter never loaded; Ollama has no feedback loop to load it into. The official model card says as much, distinguishing a backbone-only load from generation through the hybrid engine. When I gave the 8B its intended agent prompt, it mentioned the right answer to a factorial question inside its reasoning and then failed to finish within 1,024 tokens; the 14B asked for a Python tool after 324 tokens. Neither is an accuracy result. Both are a model running without the mode it was trained around.

These Ollama rates also cannot be divided into anything measured later. Runtime, precision, prompt and trajectory all change at once.

Getting the real loop onto Pascal

The repository's install example targets CUDA 12.8, and the bitsandbytes builds listed for 12.8 omit the sm_60 target these cards need. A torch 2.14 cu126 build with bitsandbytes 0.50.2 worked: NF4 double-quantized linear weights, FP32 compute and adapter, SDPA attention. Automatic placement wanted to offload the 14B output head to disk, so both sizes got an explicit even layer split across the two GPUs. The harness checks actual parameter devices rather than trusting the placement request.

I wrapped the engine in a timing harness that logs actual embedding-feedback events. A literal <latent> string in the output proves nothing; the vector injection does. Every text-only control logged zero events. Every enabled diagnostic run logged real ones with natural exits.

The first native numbers were fixed 128-step windows on the earlier prompts. They showed the cost structure: at 8B, hybrid mode ran about 12.2 total steps per second but only 5.6 visible tokens per second, because 69 of the 128 steps were latent. The control ran 12.3 tok/s. Nothing in a fixed window says whether a task would have finished sooner, because nothing finishes. This is where a tok/s comparison bottoms out.

The experiment that could answer the question

What was missing was a natural completion, an answer I could check without reading the model's reasoning, and an equal work budget for both modes. The protocol was frozen before measurement:

• Six tool-free integer questions (pencil inventory, trailing zeros of 100!, 7^222 mod 13, a lattice-path count avoiding a point, three-digit multiples of 7, a two-congruence search), each with an answer computed independently in plain Python.
• Three paired seeds (42, 43, 44), temperature 0.6, top-p 0.95, top-k 20.
• Two modes with the same checkpoint, prompt, precision, placement, sampling and a budget of 4,096 total steps, where a step is a discrete token or an actual latent step. The control masks only latent entry.
• An external, exact grader. Generation stops at a valid closed <answer> block, at EOS, or when the budget runs out. A correct number appearing in reasoning does not count.

Two sizes × six questions × three seeds × two modes is 72 measured requests, plus four excluded warmups.

Two disclosures. First, the initial 8B run shared the host with unrelated Chrome and ffmpeg work; one-minute load hit roughly 97–116 on 12 logical CPUs and a single trajectory's step rate fell from 12.93 to 3.83 steps per second mid-run. The GPUs were free; the timings were still worthless. I stopped that run, kept its ten finished requests as a separate historical file, and reran everything on a quiet host with external load sampling every ten seconds. Second, the grader was clarified early, within the first three measured requests and before any paired speedup existed: a complete answer block holding a bare integer counts, and an EOS response is graded with the released answer extractor at the actual EOS event. The original strict \boxed{} grades are preserved beside the semantic ones and reported below. No generation or timing setting changed.

Results

Mode-level numbers use all 18 attempts per mode. Paired latency uses only pairs where both modes were correct, so the two sizes' latency cohorts differ and are not comparable to each other.

Across all 72 attempts there were 43 correct completions, four wrong completions and 25 budget failures. The failures stay in every denominator above.

The 14B row needs care. Its both-correct medians are 43.40 s and 34.18 s, and dividing them gives 1.27×. That is a ratio of medians. The median of the five per-pair ratios is 0.9999, which rounds to 1.00×, and only two of those five pairs were faster with HybridLink. The 1.56× aggregate is real too, but it is driven by the two multiples pairs, at 4.51× and 1.55×. Report the median of ratios as the headline and show the others as what they are.

The all-attempt row is the one I would want before paying for anything. It sums wall time over all 18 attempts per mode, including the ones that ran out of budget. Enabled mode spent less total time at both sizes while producing more correct answers. It is elapsed time on one fixed set, not GPU-hours, energy or money.

Wins and losses, by run

Paired runs are the honest unit here, so these are individual pairs with their run IDs.

• 8B modular, seed 43: 155.12 s text-only, 64.68 s enabled, both correct. 2.40×, and the kind of pair the median is made of.
• 8B modular, seed 42: same question, 163.25 s → 188.93 s, both correct. 0.86×. One seed apart from the pair above.
• 8B inventory, seed 44: a simple arithmetic question. Text-only answered correctly in 28.80 s; enabled mode exhausted its 4,096-step budget after 290.72 s of wall time. A successful-pairs chart cannot show this run; the accuracy and all-attempt rows can.
• 14B multiples, seed 42: 117.65 s → 26.07 s, both correct, 4.51×. The largest ratio observed, and a selected example, not a typical one.
• 14B inventory, seed 42: 23.66 s → 37.02 s, both correct, 0.64×. Enabling feedback made a short problem slower.
• 14B congruences, all three seeds: text-only 0/3, enabled 3/3. A reliability gain with no finite paired speedup at all, because there is nothing to pair against.
• 14B factorial, seed 44: 43.40 s → 43.41 s. The enabled run chose zero latent steps; the engine permits latent entry, it does not force it. One of the five 14B pairs is therefore a near-identical trajectory, and it stays in the enabled-mode analysis.

The four wrong completions were all completed, wrong integers: 26 for 17, 9 and 22 for 12, 70 for 17. Some outputs also asserted code-like calculations. No code ran; the tool executor was empty. I am not quoting any of the reasoning as a solution, and the latent spans are vectors, not recoverable text.

Why tokens per second pointed the wrong way

Total steps per second barely changed, as you would expect if a latent step costs about what a token costs. Discrete tokens per second fell, because the denominator is the same generation time and some of it now holds latent work. Watch a stream and the enabled model looks slower. On the eight successful 8B pairs, median total generation work fell from 2,193 steps to 967.5, and completion latency fell with it. The useful gain was a shorter route to the answer, not a faster emitter.

That is as far as the evidence goes on mechanism. Fewer steps to a correct answer is observed. Why the trajectory shortened, and whether a latent step carries more reasoning than a token, is not something this experiment can see.

Time to first reasoning text did not change because it is almost entirely prefill plus the first few decode steps, and the latent path only engages once the model chooses to enter it. The boundary at which an answer block starts did move, but that quantity was observed for unequal numbers of runs per mode, so it is not in the table.

Where I would draw the line

I'd evaluate HybridLink for batch reasoning work where the final answer is checked and total time to a correct result is the cost, and I'd measure it the same way: paired seeds, an external grader, failures kept in the denominator, latent steps counted as compute. On this set that is where the gain lived, at both sizes, in different forms.

I would not adopt it for anything where first-token latency is the experience, on the strength of this or any similar data. First reasoning text was flat, and interactive use was not task-evaluated here.

I'd choose six real questions from the intended workload over any throughput number before deciding either way.

Limits

1. Six hand-authored questions and three correlated seeds. The 72 attempts are not 72 independent tasks; there is no confidence interval or significance claim.
2. Both modes use the same hybrid-trained weights and the same latent-instructing prompt. Masking entry may handicap the policy. This isolates a runtime switch, not the training method against a matched text-only model.
3. NF4/FP32 on Pascal, a custom tool-free prompt and the released 4/200 latent-span limits differ from the paper's H100, agentic, 16/256 setup. The paper's gains are its authors' claims.
4. Mode order was balanced overall but fixed per question across its three seeds.
5. Paired latency is conditional on both modes succeeding. Read it beside the accuracy and all-attempt rows every time.
6. The released engine still computes transformer and vocabulary logits in latent mode. Nothing here shows a free latent step or an eliminated output head. One footnote: the paper describes a nearest-embedding readout for latent exit, while the tested source uses the logits' argmax. Observed, not tested for impact, and not a bug claim.
7. No coding, chat, summarization or tool-agent task, no concurrency, energy, cost or modern-GPU replication was run.

The quiet-host rerun had 1,287 load samples with a one-minute load median of 1.90 and maximum of 7.03 on 12 CPUs, and no request crossed the predefined contention flag. That rules out the earlier disaster; it does not prove an idle box.

Data

The per-request, per-pair and per-mode exports are published with this post, with a methods README covering definitions and the grading clarification: completion-runs.csv, completion-pairs.csv, completion-modes.csv. Model outputs and per-step event logs are not published. The upstream code is pinned at commit 1b41cebb of togethercomputer/HybridAgent; the checkpoints are the 8B and 14B Agentic releases at the revisions I ran. My benchmark harness is local and unpublished.