Transformers
Source reports covering Transformers. Stories will appear when analysis and community reactions are ready.
12 reports · page 1 of 1
Connected topics are temporarily unavailable. You can still browse stories below.
Source reports
DeepSeek-v4.1-Flash-1017
tool
4 min read
DeepSeek has released DeepSeek-V4.1-Flash-1017, a minor update to DeepSeek-V4.1-Flash that it says improves reasoning, coding, agentic-task and visual-benchmark performance while reducing execution time and output length. The model is available on Hugging Face under the MIT License and can run through Transformers, vLLM, SGLang, Docker Model Runner, or local inference tools. DeepSeek reports scores of 42.6 on HLE without tools, 42.3 on Terminal-Bench 4.0, 77.0 on DeepSWE v1.1, 29.1 on ProgramBench, 68.2 on NL2Repo-Bench, and 60.5 on Automation-Bench v1.0.6. These exceed the prior Flash model’s reported scores of 36.8, 31.2, 74.2, 20.3, 65.4, and 54.8 on those benchmarks, respectively. On tool-enabled visual tests, it scored 83.0 on Chartography, 92.5 on BabyVision, and 51.0 on ZeroBench-main. The release does not include a Jinja chat template. DeepSeek provides a Python reference encoder and deepseek-recipe, Rust libraries with Python bindings for converting API-style requests into DeepSeek V4 and V4.1 prompt formats and parsing streamed or complete outputs. Recommended settings include temperature and top_p of 1.0, a one-million-token context window, and at least 256,000 maximum output tokens.
Nvidia Nemotron 3.5 Lightning
tool
39 min read
NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4 is available for use with libraries such as Transformers for tasks like text generation. Users can implement this model by importing the pipeline from the Transformers library and specifying the model for text generation tasks.
An SLM trained on $8 ESP32-S3
tool
12 min read
Carloscodix/qapla is a char-level transformer model trained from scratch on an $8 ESP32-S3 chip, capable of running the full training loop with backpropagation implemented in C. This project demonstrates that training a model does not always require expensive GPUs or datacenters, as a low-cost microcontroller can suffice.
Kimi K2.7-Code: open-source coding model with better token efficiency
tool
9 min read
moonshotai/Kimi-K2.7-Code can be utilized with libraries like Transformers for image-text-to-text tasks. A high-level pipeline can be created using the command `pipe = pipeline("image-text-to-text", model="moonshotai/Kimi-K2.7-Code", trust_remote_code=True)`.
Transformers Are Inherently Succinct
research
1 min read
Transformers exhibit inherent succinctness in their representations, enabling efficient processing of information. This research has been recognized as one of three outstanding papers for ICLR 2026.
Do transformers need three projections? Systematic study of QKV variants
research
2 min read
Transformers utilize a query, key, and value (QKV) attention formulation that is crucial for AI tasks. The study investigates the individual contributions of these three projections and the effects of omitting any of them.
Soul Player C64 – A real transformer running on a 1 MHz Commodore 64
tool
5 min read
A transformer model with approximately 25,000 parameters is implemented on an unmodified Commodore 64, utilizing hand-written 6502/6510 assembly. This 2-layer decoder-only architecture features real multi-head causal self-attention, softmax, and RMSNorm, and can be loaded from a floppy disk.
Attention Residuals
tool
3 min read
Attention Residuals (AttnRes) serves as a drop-in replacement for standard residual connections in Transformers, allowing each layer to selectively aggregate earlier representations. It includes two variants: Full AttnRes, where each layer attends over all previous outputs, and Block AttnRes, which groups layers into blocks to reduce memory usage from O(Ld) to O(Nd).
Building a Minimal Transformer for 10-digit Addition
tool
1 min read
A minimal transformer model has been developed to perform 10-digit addition tasks. The model demonstrates the ability to learn and execute arithmetic operations effectively.
FORTH? Really!?
opinion
3 min read
FORTH and associative/applicative languages may be more suitable for transformer architectures than traditional top-down problem-solving methods. Generating outputs before their constituent parts could enhance the effectiveness of large language models.
Attention at Constant Cost per Token via Symmetry-Aware Taylor Approximation
2 min read
Self-attention mechanisms in Transformers typically incur costs that increase with context length, leading to higher demands for storage, compute, and energy. A new method using symmetry-aware Taylor approximation aims to maintain constant cost per token for self-attention, potentially alleviating these resource demands.
David Patterson: Challenges and Research Directions for LLM Inference Hardware
2 min read
Large Language Model (LLM) inference faces significant challenges primarily related to memory and interconnect issues rather than compute power. The autoregressive Decode phase of Transformer models distinguishes LLM inference from training, complicating the process.