Fine Tuning a Local LLM to Categorize Questions
<p>As a fun personal project, I have been working on a chatbot for answering general questions about my household on anything from maintenance questio...
LLM Engineering Pattern
Summary
A pattern for LLM classification at scale: let a small model hallucinate plausible fake category labels, then resolve them to real taxonomy entries via embedding similarity. This avoids shipping large schema lists and enables cheaper models.
Highlights
- The classic approach of using structured outputs with a giant Literal of 500+ valid categories is expensive and hits token limits
- The pattern: ask a cheap LLM to invent plausible fake classifications, then resolve to real ones via embedding similarity against pre-computed taxonomy embeddings
- Enables classification with smaller, cheaper models at scale without shipping large schema lists
- Demonstrated with the Wayfair WANDS e-commerce dataset where queries like 'wood coffee table' need to map to hundreds of hierarchical categories
- The hallucination prompt uses few-shot examples of the taxonomy format to guide the model toward structurally plausible outputs
auto-generated
Context
Audience
ML engineers and developers building LLM-powered classification systems, particularly those working with large taxonomies or e-commerce product categorization at scale
DomainMachine Learning / LLM Engineering
Formatblog post with code examples
Accessfree online
Source Overview
This article describes a pattern where a small LLM generates plausible but fake category labels for a query instead of being constrained to a fixed taxonomy via structured outputs.
The hallucinated labels are resolved to real taxonomy entries by computing embedding similarity against pre-computed embeddings of valid classifications. The approach eliminates the need to ship large schema lists to the model and enables classification with cheaper models at scale.
Topics
Related
Structured Outputs / Constrained DecodingEmbedding Similarity SearchWayfair WANDS DatasetPydantic response parsingFew-shot prompt engineeringProduct categorization in e-commerce
