Tool calling, open source, and the M×N problem
Tool calling with closed-source models is seamless. You pass a list of functions to the API, the model calls them, you get structured JSON back.
AI Inference Engineering
Summary
vLLM 0.31 introduced constrained decoding via XGrammar, guaranteeing 100% tool-call schema accuracy for Kimi and DeepSeek models, solving the 92% accuracy gap in earlier versions for agentic workloads.
Highlights
- vLLM 0.31 added constrained decoding as a server-side flag, enabling guaranteed schema adherence for tool calls
- Prior to this, vLLM 0.28 achieved only 92% tool-call schema accuracy with Kimi K2.6, meaning 1 in 25 calls failed
- The technique uses the XGrammar library to block schema-invalid tokens at each generation step
- Schema accuracy is critical for agentic workloads where multi-turn sessions compound errors from nonsensical context
- UCloud had been maintaining a custom vLLM fork to achieve 100% accuracy before this feature was upstreamed
auto-generated
Context
Audience
AI engineers and developers serving open-weight models in production, particularly those building agentic applications
DomainLLM Inference & Serving
Formattechnical blog post
Accessfree online
Source Overview
vLLM 0.31 added constrained decoding, guaranteeing 100% tool-call schema accuracy for Kimi and DeepSeek models.
Prior versions like 0.28 achieved only 92% accuracy with Kimi K2.6. The technique uses the XGrammar library to block schema-invalid tokens at each generation step.
Topics
Related
vLLMXGrammarKimi K2.6DeepSeekConstrained DecodingJSON Schema validation
