# HybridLink on/off completion experiment: public data

Companion data for the cutler.sg article "HybridLink on Two Pascal GPUs: Tokens per Second Was the Wrong Metric". Experiments ran on 24 October 2026 (final run 22:54 SGT 24 Oct to 02:29 SGT 25 Oct). This is a sanitized export of one exploratory local experiment, not a benchmark release.

## Files

- `completion-runs.csv`: one row per measured request (72 rows). Columns: `run_id`, `size` (8b/14b), `task`, `seed`, `mode` (`text-only` control or `hybrid` enabled), `correct` (semantic grade), `strict_correct` (original strict `\boxed{}` grade), `expected`, `parsed_integer`, `failure`, `end_reason` (`answer_complete`, `eos`, `budget`), `completion_basis`, timing columns in seconds (`prefill_s`, `first_sampled_token_s`, `first_reasoning_text_s`, `first_answer_text_s`, `answer_complete_s`, `wall_s`, `generation_s`), work counts (`prompt_tokens`, `text_tokens`, `latent_steps`, `generation_steps`), rates (`text_tokens_per_s`, `generation_steps_per_s`, `latent_phase_s`, `latent_phase_steps_per_s`), and `source` + `json_pointer`, which address the record inside the original raw file in the local benchmark checkout (not a public URL).
- `completion-pairs.csv`: one row per task/seed pair (36 rows) with off/on correctness, end reasons, completion seconds, wall seconds and `speedup` = `off_complete_s / on_complete_s`. Blank means no eligible ratio (one or both modes failed), never zero.
- `completion-modes.csv`: four per-mode summaries (size × mode) including `correct`, `total_wall_s`, `failed_wall_s`, medians and the paired cohort values.

Blank CSV values mean unavailable. Four warmups (seed 0, 256-step cap) are excluded from every file. Model outputs, latent vectors and per-step event logs are not published here.

## Setup

- Models: `togethercomputer/HybridAgent-8B-Agentic` (revision `13a5c61b`) and `HybridAgent-14B-Agentic` (revision `1b726e32`), each with its own released `adapter.pt`. Engine: upstream `togethercomputer/HybridAgent` at commit `1b41cebbd2388d083ee1820af421b7f9e24affb5`, unmodified; a local harness wrapped it for timing.
- Hardware: 2 × NVIDIA TITAN X (Pascal, 12 GB each), driver 565.57.01, i7-8086K (12 logical CPUs), ~62 GiB RAM.
- Precision/runtime: NF4 double-quantized linear weights, FP32 compute/unquantized modules/adapter; torch 2.14.0+cu126, Transformers 5.13.0, bitsandbytes 0.50.2, SDPA attention; explicit even layer split across both GPUs for both sizes; batch 1, sequential requests, fresh KV cache per request.
- Prompt: one custom tool-free system prompt instructing `<think>` reasoning, optional `<latent>` spans inside it, and a final `<answer>\boxed{N}</answer>`. Tools disabled and capped at zero; no generated code or search call was executed.
- Sampling: temperature 0.6, top-p 0.95, top-k 20, repetition penalty 1.0; seeds 42/43/44 paired across modes. Work budget 4,096 total generation steps per request (discrete tokens + actual latent steps). Latent span limits 4/200 (released-code defaults).
- Modes: `hybrid` lets the model enter latent mode; `text-only` is the same checkpoint, prompt and settings with latent entry masked. This is a runtime ablation, not a comparison against a separately trained text model.
- Order: task order rotates by seed and pair order alternates (nine pairs start in each mode per size), but each question keeps the same mode order across its three seeds.

## Questions and independently checked answers

| task | question | answer |
| --- | --- | --- |
| inventory | 7 boxes × 24 pencils, sell 89, donate 35; how many remain? | 44 |
| factorial | Trailing zeros of 100! | 24 |
| modular | Remainder of 7^222 divided by 13 | 12 |
| lattice | Paths (0,0)→(4,3) by unit right/up steps avoiding (2,1) | 17 |
| multiples | Three-digit positive integers divisible by 7 | 128 |
| congruences | n in [50,150], n ≡ 3 (mod 7), n ≡ 5 (mod 11) | 115 |

## Grading, including the early clarification

The generator stops at the first complete valid `<answer>…</answer>` block outside reasoning, at actual EOS, or at the budget. It never stops because the expected number appears in reasoning.

The original raw grader (`strict_correct`) required exactly one signed-integer `\boxed{}` inside a valid closed answer block. Early in the run, after the warmups and the first measured hybrid request and before any paired speedup existed, two format cases were clarified: a complete answer block containing only a bare integer is accepted, and an EOS response is graded with the released `extract_answer` fallback (last boxed value) with the completion time set to the actual EOS token event. This semantic grade (`correct`) is the headline grade in the article. Strict counts are reported alongside: 8B 5/18 text-only and 10/18 hybrid; 14B 5/18 and 5/18. Wrong answers and budget truncations fail under both grades. No generation or timing parameter changed.

## Timing definitions

- Time starts before engine initialization/prefill; both GPUs are synchronized. Native token-availability timings, no HTTP or queue.
- `first_reasoning_text_s`: first non-whitespace readable text, skipping control anchors. Not a final answer and not HTTP TTFT.
- `answer_complete_s`: emission of the valid closing answer tag, or the actual EOS event for the extractor fallback. Absent for budget failures.
- `wall_s`: whole request including the final redundant forward and cleanup; present for failures.
- `generation_steps_per_s` = (text_tokens + latent_steps) / generation_s. `text_tokens_per_s` uses the same duration, so it falls when time is spent on latent steps.
- Paired speedup is defined only when both modes are correct: `off_complete_s / on_complete_s`. Headline = median of per-pair ratios; this is not the ratio of medians.
- All-attempt time = sum of `wall_s` over all 18 attempts per mode, failures included; excludes model load, warmups and between-request bookkeeping. It is elapsed time, not energy or money.

## Host conditions

External sampling every ~10 s: 1,287 samples, one-minute load min/median/max 0.85/1.90/7.03 on 12 logical CPUs, CPU busy median 15.2%, 0/72 trial windows above the predefined load1 > 12 flag, every request covered. This rules out the severe contention seen in an earlier partial run (which is excluded entirely); it does not prove zero background activity or fixed clocks.

## Limits

Six hand-authored questions and three correlated seeds; descriptive only. Same hybrid-trained weights in both modes. NF4/FP32 on Pascal with a tool-free prompt and 4/200 spans differs from the paper's H100 agentic setup and 16/256 spans. Mode order fixed per question across seeds. Latent steps still run the transformer and vocabulary head in the released engine. No energy, cost, concurrency, modern-GPU or separately trained baseline measurement was made.