tokenization
Latest stories covering tokenization, with summaries and links to related coverage.
4 articles · page 1 of 1
Connected topics
GigaToken: ~1000x faster Language model tokenization
tool
12 min read
Gigatoken enables language model tokenization at speeds of GB/s, making it approximately 1000 times faster than HuggingFace's tokenizers. It serves as a drop-in replacement, supports a wide range of CPU hardware, and is compatible with multiple commonly used tokenizers.
The real prices of frontier models
research
11 min read
Anthropic's newest tokenizer, Sonnet 5, generates approximately 30% more tokens from the same code compared to its previous version. For identical files, it produces 1.36 to 1.73 times the token count of GPT, with TypeScript resulting in the highest increase at 1.73 times.
Price per 1M tokens is meaningless
opinion
4 min read
Comparing AI models by price per 1M tokens can be misleading due to differences in tokenization methods among various labs. Each tokenizer affects how text is divided into tokens, which can significantly influence overall costs.
Claude Opus 4.7 costs 20–30% more per session
research
1 min read
Claude 4.7's new tokenizer uses 1.47 times more tokens than previous versions, exceeding the documentation estimate of 1.0–1.35x. This increase impacts the cost of processing content.