Latest stories
Story · 1 report
Scaling Hierarchical Networks
Cartesia Research reported that its hierarchical networks, or H-Nets, learn to group raw bytes into dynamic chunks inside a language model rather than relying on a fixed Byte Pair Encoding tokenizer. The October 2026 results compare byte-level H-Nets with tokenized Transformer baselines under matched data and compute settings. In a one-stage configuration designed as a learned replacement for tokenization, Cartesia said H-Nets were 1.28 times more compute-efficient on in-domain validation loss and about twice as efficient on downstream evaluation suites. Across 15 programming languages in the training mix, the reported median efficiency gain was 2.04 times; across 10 out-of-domain languages, it was 3.10 times, reaching 5.37 times on Prolog. A larger two-stage byte-level H-Net used a 40-layer Transformer core with Mamba-2-based encoder and decoder stages and was compared with Hugging Face’s 3.08-billion-parameter SmolLM3. Cartesia reported that its 3.67-billion-parameter H-Net reached SmolLM3’s final average benchmark score after roughly 1 trillion training tokens, versus about 11 trillion tokens for SmolLM3, while matching compute per input byte within 2%. Cartesia also reported dynamic, context-dependent chunking behaviors, including grouping recurring words, formats, code identifiers, and motifs. Preliminary experiments extended the architecture to DNA and pixel-space image generation; Cartesia reported a 6–7 times compute gain for DNA sequence modeling.
cartesia.ai
🔥🔥🔥🔥🔥2 min summary
10/28/2026
No more articles to load
