Deep learning
Latest stories covering Deep learning, with summaries and links to related coverage.
1 stories · 1 reports · page 1 of 1
The topic overview is temporarily unavailable. You can still browse coverage below.
Connected topics are temporarily unavailable. You can still browse stories below.
Latest stories
DeepSeek just open-sourced its inference engine
tool
2 min read
DeepSeek has open-sourced DeepInfer, the high-performance inference framework used in its large-scale production services. The release supports offline inference for DeepSeek-V4.1-Flash and includes optimized kernels and model and sampler runtimes. Its parallelism features include attention data parallelism, balanced context parallelism, mixture-of-experts expert parallelism, and expert parallel load balancing. DeepInfer uses FlashMLA, DeepGEMM, DeepEP, TileKernels, DeepSelect, and custom TileLang kernels. The framework supports DSpark and confidence-scheduled verification to accelerate decoding with draft tokens, decoder sliding-window-attention bounded replay to reduce prefill work, and asynchronous scheduling that overlaps CPU batch scheduling with device execution. It can provide batch-invariant inference and deterministic sampling on NVIDIA hardware, and it can constrain outputs with regular expressions, JSON schemas, or EBNF grammars. Supported hardware includes NVIDIA SM100 GPUs and Huawei Ascend 950 NPUs. DeepInfer requires Linux, GCC 13 or newer, and Python 3.12, plus CUDA 13 for NVIDIA or CANN 9.2 for Ascend. A complete serving system still requires components such as a KV-cache system, prefill/decode disaggregation, and load balancing across data-parallel replicas. DeepInfer is available under the Apache License 2.0.