Fixing Tool-Call Accuracy in vLLM
Tool call accuracy is an important part of serving open-weight AI models. When your tool calls aren’t accurate, the model builds up on nonsensical con...
Technical Blog Post
Summary
Open-source LLMs use incompatible wire formats for tool calling, forcing each inference engine to independently write custom parsers per model family. Generic parsers fail because formats are unconstrained training-time decisions. The post argues for declarative specifications as a solution.
Highlights
- Closed-source models hide the wire format; open-source models expose it, forcing engines to parse model-specific tool call encodings
- Each inference engine (vLLM, SGLang, TensorRT-LLM) independently writes custom parsers for every model family it supports
- Generic parsers fail because wire formats are unconstrained training-time decisions with no shared convention across model families
- Edge cases like reasoning token leakage and non-standard argument serialization (e.g., arg_key/arg_value pairs) break heuristic parsing
- The proposed solution is declarative specifications consumable by both grammar engines and output parsers, analogous to chat templates
auto-generated
Context
Audience
ML engineers and developers building tool-calling pipelines with open-source LLM inference engines
DomainLLM Infrastructure
Formattechnical blog post
Accessfree online
Source Overview
Open-source language models use incompatible wire formats for tool calling, forcing each inference engine (vLLM, SGLang, etc.) to independently write custom parsers for every model family—creating an M×N multiplication of duplicate effort.
Generic parsers fail because model formats are unconstrained training-time decisions with no shared convention, leaving edge cases like reasoning token leakage and special token stripping unhandled. The solution requires declarative specifications that both grammar engines and output parsers can consume, similar to how chat templates standardized prompt formatting in Hugging Face.
Topics
Related
vLLMSGLangllama.cppHugging Face chat templatesGrammar enginesStructured output generation
