Fixing Tool-Call Accuracy in vLLM
Tool call accuracy is an important part of serving open-weight AI models. When your tool calls aren’t accurate, the model builds up on nonsensical con...
ML Inference Engine
Summary
An MLX-based inference engine for Apple Silicon that evaluates multi-field JSON schemas in parallel via KV-cache broadcasting, achieving 5.6x to 7.0x latency reduction over autoregressive decoding with 100% schema validity.
Highlights
- KV-cache broadcasting enables all schema fields to be evaluated in a single parallel pass instead of sequential token-by-token generation
- 5.6x to 7.0x latency reduction on Apple Silicon M4 Max compared to autoregressive baselines (e.g., 270ms vs 1,900ms for 28-field triage)
- 100% schema validity guaranteed by construction, eliminating syntax errors, field omissions, and hallucinated keys
- Supports up to 255 enum choices per field with calibrated confidence scores for structured extraction, decision routing, and categorical classification
- Includes a Python SDK, M4 Max performance benchmarks, and a live side-by-side web visualizer on Hugging Face Spaces
auto-generated
Context
Audience
ML engineers and LLM practitioners working on Apple Silicon who need low-latency structured output generation for production pipelines such as classification, routing, or extraction tasks
DomainMachine Learning / LLM Inference
Formatopen-source project with documentation and live demo
Accessopen source on Hugging Face
Source Overview
An MLX-based inference engine for Apple Silicon that evaluates multi-field JSON schemas in parallel via KV-cache broadcasting, achieving 5.6x to 7.0x latency reduction over autoregressive decoding with 100% schema validity.
Supports structured extraction, decision routing, and categorical classification with up to 255 enum choices per field and calibrated confidence scores. Includes a Python SDK, M4 Max performance benchmarks, and a live web visualizer for side-by-side comparison.
Topics
Related
MLX frameworkQwen 2.5 modelsConstrained decodingStructured output generationApple Silicon NPU optimizationKV-cache optimization
