Themata.AI
Follow coverage
Themata.AI
AI agentsMachine learningData efficiencyHierarchical networks

Report · cartesia.ai

Scaling Hierarchical Networks

From Tokens to Bytes I: Scaling Hierarchical Networks | Cartesia

cartesia.ai

Source published October 28, 2026

Summary: 2 min read

🔥🔥🔥🔥🔥

49/100

Summary

Cartesia Research reported that its hierarchical networks, or H-Nets, learn to group raw bytes into dynamic chunks inside a language model rather than relying on a fixed Byte Pair Encoding tokenizer. The October 2026 results compare byte-level H-Nets with tokenized Transformer baselines under matched data and compute settings. In a one-stage configuration designed as a learned replacement for tokenization, Cartesia said H-Nets were 1.28 times more compute-efficient on in-domain validation loss and about twice as efficient on downstream evaluation suites. Across 15 programming languages in the training mix, the reported median efficiency gain was 2.04 times; across 10 out-of-domain languages, it was 3.10 times, reaching 5.37 times on Prolog. A larger two-stage byte-level H-Net used a 40-layer Transformer core with Mamba-2-based encoder and decoder stages and was compared with Hugging Face’s 3.08-billion-parameter SmolLM3. Cartesia reported that its 3.67-billion-parameter H-Net reached SmolLM3’s final average benchmark score after roughly 1 trillion training tokens, versus about 11 trillion tokens for SmolLM3, while matching compute per input byte within 2%. Cartesia also reported dynamic, context-dependent chunking behaviors, including grouping recurring words, formats, code identifiers, and motifs. Preliminary experiments extended the architecture to DNA and pixel-space image generation; Cartesia reported a 6–7 times compute gain for DNA sequence modeling.

Key Takeaways

  • H-Nets replace fixed text tokenizers with learned, end-to-end chunking of raw bytes into higher-level representations.
  • Cartesia reported that a controlled one-stage H-Net achieved 1.28 times better compute efficiency than a BPE Transformer on in-domain validation loss and about twice the efficiency on downstream evaluations.
  • Cartesia reported median H-Net compute-efficiency gains of 2.04 times on programming languages in its training mix and 3.10 times on 10 programming languages outside that mix.
  • In Cartesia’s comparison with SmolLM3, a two-stage H-Net reportedly reached the baseline’s final average benchmark score after about 1 trillion tokens rather than 11 trillion tokens, with compute matched per input byte.
  • Cartesia reported that H-Nets dynamically adapt chunk boundaries during inference and showed preliminary results in DNA modeling and pixel-space vision.

What the discussion said

Commenters spent less time on the paper’s headline than on its wager: replace a fixed tokenizer with a hierarchy that learns larger units from primitive input atoms. Several readers argue that byte-level input should not be romanticized as intrinsically natural. Bytes are themselves a contingent encoding layer, especially for Unicode, and the architecture only needs some irreducible starting representation before it composes useful higher-level patterns. The key attraction is that the model could discover multi-character, morphological, or cross-modal structure rather than inheriting BPE’s frequency-driven segmentation mistakes. Supporters see that as more than a language-model tweak. A shared scheme that starts from raw-ish inputs and learns abstraction hierarchically could unify text and images while preserving transfer benefits associated with byte-level processing. They reject the claim that hierarchical networks necessarily forgo compression: learned patterns can serve the same role as multi-character tokens without being frozen into a vocabulary. A lighter note of enthusiasm greeted the unusual examples used to illustrate learned tokenization. Skeptics focus on the economic reality of generation. BPE exists because recurring text patterns can be packed and decoded efficiently; processing small atoms risks losing that advantage and making inference much slower. With frontier systems increasingly leaning on long speculative-decoding runs, they argue the proposal needs hard measurements showing it can compete, not just a promise of learned token boundaries. They suspect its practical niche may be narrow, perhaps structured data where conventional tools already fit well.

Where opinion split

The central fight is whether learned hierarchical segmentation can retain tokenizer-like efficiency while escaping BPE’s semantic blind spots. Supporters say it can learn multi-character structure dynamically, avoid arbitrary frequency-based boundaries, and extend naturally across modalities; skeptics say fixed tokenizers exploit redundancy so effectively that atom-level processing will lose badly on ordinary coding and English generation unless experiments prove otherwise.

Read original article

Coverage & discussions

Hacker News · cartesia.ai

Scaling Hierarchical Networks ↗

Community discussions

Hacker News discussion ↗

Reported comments: 12 · 3 saved in sample

Analysis ready

Community Sentiment

Mixed

Positives

  • Learning segmentation inside the network could replace BPE’s frequency-driven splits with units that better track morphology and syntax, removing a persistent source of arbitrary linguistic distortion.
  • A hierarchy built from simple input atoms can still discover recurring multi-character patterns, preserving compression without permanently hard-coding a vocabulary.
  • Using one learned abstraction mechanism for text and images could make multimodal transfer less dependent on incompatible, hand-designed preprocessing pipelines.
  • The examples of learned structure made the approach feel tangible rather than purely architectural, giving readers confidence that dynamic token formation may expose useful patterns.

Concerns

  • Bytes are not a uniquely natural foundation for intelligence; Unicode makes them another arbitrary encoding convention, so byte-level claims can overstate what the representation buys.
  • Fixed tokenizers aggressively exploit repeated patterns in English and code, and a hierarchy that begins with tiny atoms may pay a steep decoding and compute penalty.
  • Long speculative decoding raises the performance bar: an approach that emits or processes finer-grained units must demonstrate competitive throughput, not merely theoretical flexibility.
  • For highly structured data, critics doubt this architecture opens a compelling new use case when language models can already delegate tabular work to SQL or dataframe tools.

Related Articles

LLM Neuroanatomy II: Modern LLM Hacking and hints of a Universal Language?

LLM Neuroanatomy II: Modern LLM Hacking and Hints of a Universal Language?

Mar 24, 2026

Hypernetworks: Neural Networks for Hierarchical Data

Hypernetworks: Neural Networks for Hierarchical Data

Feb 5, 2026

Understanding LLM Inference Engines: Inside Nano-vLLM (Part 1) - Neutree Blog

Nano-vLLM: How a vLLM-style inference engine works

Feb 2, 2026

Neural Networks Are Already Clean

Neural Networks Are Already Clean

Oct 15, 2026

Alexi Gladstone | Explorative Modeling -- Unlocking a Third Pretraining Axis and End-to-End Generation

Explorative modeling: Train on the best of K guesses

Aug 1, 2026