Report · ubicloud.com
Fixing Tool-Call Accuracy in vLLM
ubicloud.com
Source published October 26, 2026
Summary: 2 min read
56/100
Summary
Ubicloud enabled constrained decoding in vLLM to enforce valid tool-call schemas for Kimi and DeepSeek models, reporting an increase in K2VV ToolCall benchmark schema accuracy from 92% to 100%. The company had observed that Kimi K2.6 served through vLLM 0.28 produced the expected tool-call schema 92% of the time, meaning roughly two of every 25 calls were malformed. Invalid calls can add error messages to an agent’s context, reduce output quality, and require retries that increase costs. Tool calls require an LLM to emit structured arguments matching a declared JSON schema, so an agent harness can invoke functions such as web-page fetching. Constrained decoding prevents the inference engine from selecting tokens that would violate the required tool-call structure or JSON schema during generation. vLLM uses structural tags from XGrammar to restrict valid tokens at each stage, including tool names, call indexes, and arguments. Although vLLM added constrained decoding in version 0.25, it does not enable strict tool decoding by default for all requests because support is model-specific and can cause problems for some models. vLLM later added the server-side --tool-strict-level flag in version 0.31. Moonshot AI recommends constrained decoding for Kimi, and Ubicloud deployed it after validating 100% schema accuracy across its served model versions.
Key Takeaways
- Ubicloud reported that enabling constrained decoding in vLLM raised Kimi and DeepSeek tool-call schema accuracy from 92% to 100% on Moonshot AI’s K2VV ToolCall benchmark.
- The K2VV ToolCall benchmark contains 2,000 requests and measures trigger similarity and schema accuracy, or the share of generated calls that conform to their tool schemas.
- Constrained decoding restricts token selection during generation so that an LLM cannot emit syntactically invalid tool-call structures or schema-invalid JSON.
- vLLM’s --tool-strict-level flag, released in version 0.31, lets deployments enable tool-call strictness; the feature is not universally enabled by default because model support varies.
- Tool-call correctness depends on inference-engine settings as well as the underlying model, particularly in multi-turn agent workloads.
What the discussion said
The thread treated reliable tool use as an agent-systems requirement, not a cosmetic formatting improvement. Commenters stressed that a few malformed calls become catastrophic when an agent chains dozens or hundreds of sequential actions: even a 95% per-call success rate makes a long workflow unlikely to finish cleanly. Constrained decoding therefore drew broad support as the practical way to force calls into the active schema, matching the reliability users associate with leading hosted providers and potentially opening serious multi-tool workloads to vLLM deployments. The enthusiasm came with important caveats. Several readers argued the post blurred a configuration and versioning problem into a dramatic vLLM repair, while the authors clarified that an upstream code change had in fact been needed for one model and that current releases expose the right strictness setting. Others described vLLM as powerful but brittle, with behavior changing across releases and demanding careful benchmark coverage. The deeper unresolved question is quality: grammar constraints guarantee valid syntax, but may distort the model's choices. Readers want evidence comparing constrained generation against detecting a bad call and retrying, including latency, token cost, and end-to-end task success. The conversation also saw broader promise in grammar-guided decoding for code and structured outputs, although implementation inside fast inference engines remains difficult.
Where opinion split
The sharp dispute is whether hard schema constraints are the best route to dependable tool use. Supporters argue that invalid calls cannot be tolerated in long agent trajectories, so decoding must prevent them token by token; skeptics argue that this changes the model's output distribution and a whole-call retry strategy could preserve better task quality at negligible cost. No one presented the comparative evaluation needed to settle it.
Coverage & discussions
Hacker News · ubicloud.com
Community discussions
Community Sentiment
Positives
- Hard-constraining tool-call tokens turns fragile agent chains into viable systems, because a tiny per-call failure rate explodes across hundreds of sequential actions.
- Current vLLM controls can now deliver schema-valid calls for supported models, narrowing a reliability advantage that hosted frontier providers held for years.
- Grammar-guided decoding is seen as a general structured-generation primitive, with obvious reach from tool arguments to syntactically valid code and browser-side outputs.
- Benchmarking serving configurations caught regressions that model-level tests would miss, highlighting that tool reliability belongs to the whole inference stack.
Concerns
- The article's framing left some readers thinking vLLM itself had been newly fixed when much of the immediate remedy was correct version-specific configuration.
- vLLM is praised for performance but criticized as fragile and overcomplicated, forcing substantial experimentation before constrained tool use behaves reliably.
- Schema validity is not task quality: forcing only legal next tokens may alter model behavior, and the thread lacks evidence that it beats error-feedback and retry loops.
- Tool schemas can change during a run, while chat and tool-update conventions remain unsettled, making fully constrained production tool environments harder than a static demo suggests.
Related Articles
Kimi vendor verifier – verify accuracy of inference providers
Apr 20, 2026
Smaller, faster, safer: running Kimi and GLM at scale
Aug 3, 2026
Kimi K2.7-Code: open-source coding model with better token efficiency
Jun 12, 2026
There are now 5 inference providers for abliterated DeepSeek
Oct 27, 2026
How OpenAI turned its coding tools into dots, an always-on personal assistant
Oct 27, 2026