GitHub - bilikaz/qwen38-flash-next-recipe: qwen38-flash-next recipe for solo GB10
qwen38-flash-next recipe for solo GB10. Contribute to bilikaz/qwen38-flash-next-recipe development by creating an account on GitHub.
LLM Deployment Scripts
Summary
Shell scripts to serve the unsloth/Qwen3.8-27B-NVFP4 checkpoint via vLLM in Docker on NVIDIA DGX Spark (GB10, aarch64) or RTX 6000 PRO, featuring NVFP4 quantization, 1M-token YaRN context, FP8 KV cache, and MTP speculative decoding.
Highlights
- NVFP4 4-bit quantization with FP8 KV cache for ~2x KV memory savings
- 1M-token context via YaRN extension (4x over native 262,144 tokens)
- MTP speculative decoding with 2 speculative tokens for faster inference
- Idempotent start script with health-check polling and log streaming
- Built-in Qwen3 thinking mode parsing and qwen3_coder tool calling support
auto-generated
Context
Audience
ML engineers and developers deploying large language models on NVIDIA edge or workstation hardware
DomainLarge Language Model Deployment
FormatGitHub repository with shell scripts
AccessOpen source on GitHub
Source Overview
The repo provides shell scripts to serve the unsloth/Qwen3.8-27B-NVFP4 checkpoint via vLLM in Docker on NVIDIA DGX Spark (GB10, aarch64) or RTX 6000 PRO.
It uses NVFP4 4-bit quantization, FP8 KV cache, YaRN-extended 1M-token context, and MTP speculative decoding. Three scripts handle download, server start (OpenAI-compatible API on port 8888), and stop.
Topics
Related
vLLMunsloth/Qwen3.8-27B-NVFP4NVIDIA DGX Spark GB10NVFP4 quantizationYaRN context extension
