Qwen3.8-27B - SGLang Documentation
Deploy Qwen3.8-27B with SGLang — dense hybrid GDN vision-language model with BF16/FP8/NVFP4 W4A4 checkpoints and in-checkpoint MTP, single-GPU on H200...
Machine Learning Model
Summary
A highly efficient 2-bit quantized version of the Qwen3.8-27B large language model designed to run on consumer-grade GPUs. It enables a 27B parameter model to fit within approximately 10.15 GB of VRAM, allowing for large context windows on hardware like the RTX 3090 or 4090.
Highlights
- Uses mixed 2/3-bit quantization to achieve a small 10.15 GB weight footprint
- Optimized to fit a 27B model and 64k context on a single 24 GB consumer GPU
- Maintains competitive performance in commonsense reasoning and coding compared to FP8 references
- Requires a specialized SGLang-based runtime (escha-runtime-qwen3dense) for deployment
auto-generated
Context
Audience
Machine Learning Engineers and AI Developers
DomainMachine Learning
FormatModel weights and deployment instructions
AccessOpen source
Source Overview
2-bit quantized build of Qwen3.8-27B (10.15 GB weights) that fits a 64k context on a single 24 GB NVIDIA GPU.
Benchmarked against an FP8 reference, it matches or slightly leads on commonsense reasoning and LiveCodeBench while trailing by one question on GPQA-Diamond. Served via a custom SGLang runtime exposing an OpenAI-compatible HTTP interface with configurable thinking mode and reasoning-effort levels.
Topics
Related
Qwen3.8-27BSGLangQuantizationNVIDIA CUDAPyTorchHugging Face
