← Research corpus

Wins

The arXiv tier

• 22,108 papers, about 411M composed tokens of math-preserving prose from LaTeX source: a pool of 22,775 e-prints fetched, 22,740 with usable source, 22,108 passing the quality gate. The fetch drained the pool at three seconds per request over about fifteen hours. ¹
• A de-LaTeX repair that recovered 53% of the tier. Papers under 500 bytes of prose went from 3,265 to 13; the tier from 261M to 399M tokens, by substituting included files in place and inlining only a member’s body. Two rounds, the second found by four sub-agents.²
• A quality gate that catches 79 of 79 over-expanded papers with one invariant: prose bytes over source bytes must be below 1.0. The full gate enforces a small set of thresholds, each for a failure class that was actually seen: no prose output, or under 500 bytes of it (the blank tier); yield under 5% of the prose-eligible body rejected and under 15% warned (the truncation and nested-document bugs); expansion ratio over 1.5 rejected and over 1.0 warned (the macro blow-ups); and duplicated-text over-generation rejected. Validated as a detector, not assumed: the pre-repair tier fails it at 15.1%, the repaired tier passes at 0.09% rejects, and the yield rule was itself corrected after it false-positived on papers that were mostly embedded figures. See gate on invariants. ³
• Math kept, not hollowed. The math-preserving converter produces zero placeholder tokens where the earlier one had removed 116,864 equations. That earlier hollowing was the cause of v2’s over-stripped-math descent. ⁴
• Token counts measured, not derived. The tier’s size was re-measured by proportional sampling with a mean-of-ratios estimator and an end-to-end check within 0.68%, replacing a number that had been circular. ⁵
• An OCR bake-off decided on rendered pages. For the 632 PDF-only submissions, two tools were compared against the rendered page at 300 DPI rather than by counting braces. The slower one won on math fidelity; the faster one had exited successfully having written nothing. The resulting tier is kept separate and out of math-sensitive evaluation.⁶

The composed corpora

• v3: about 4.3B external tokens across fourteen tiers, deduplicated between overlapping sources by MinHash, licence-filtered, quality-gated, and the whole thing sized to SmolLM2-style tokens per parameter. ⁷
• v5: about 40B tokens across nineteen tiers, fetched by two supervised builders on carlo, with licence attribution (CC-BY and CC-BY-SA tiers) propagated into both the licence manifest and the composed manifest. ⁸
• Composition as a workflow with a fail-closed guard. The corpus-build workflow fans out per tier (fetch, gate), barriers on deduplication, then composes, and refuses to compose at all unless the tag names the intended corpus. The old default now fails closed.⁹
• Contamination re-certification of the pinned v5 corpus against every frozen probe set, with hits held for me to rule on rather than passed. ¹⁰

Backed up, provably

• The irreplaceable 5 GB of derived arXiv prose is on three restic targets including S3, restore-verified by checksum on three files including a 60 MB shard, the day a gap audit found it backed up nowhere. ¹¹
• A second life as retrieval. The composed v5 training shards, verified byte-identical, are now also the substrate LiLM indexes, with the validation shards deliberately excluded. ¹²