GitHub - MakazhanAlpamys/Soup: Fine-tune LLMs from one YAML. Layer streaming trains an 8B model on a 4 GB laptop GPU.
Fine-tune LLMs from one YAML. Layer streaming trains an 8B model on a 4 GB laptop GPU. - MakazhanAlpamys/Soup
Open-source Language Model
Summary
GuppyLM is a small open‑source language model (~9M parameters) that pretends to be a fish, speaking in short lowercase sentences about tank life. It is trained on synthetic conversations with a simple 6‑layer transformer and can be built in minutes on a single GPU using the provided Colab notebook.
Highlights
- Trains from scratch on 60 K synthetic one‑turn fish‑themed dialogues
- Simple vanilla transformer: 6 layers, 384 hidden size, 4 096 BPE vocab, 128 token context window
- End‑to‑end training pipeline in a Colab notebook; ~5 minutes on a T4 GPU
- Pre‑trained weights and dataset available on HuggingFace for immediate use
- MIT licensed code for tokenizer, training loop, and inference
auto-generated
Context
Audience
Machine learning hobbyists, AI educators, and developers who want to experiment with tiny language models using minimal hardware
DomainMachine Learning
FormatGitHub repository with notebooks and scripts
Accessfree open source on GitHub
Source Overview
GuppyLM is a 9M parameter language model trained from scratch on 60,000 synthetic single-turn conversations across 60 topics, simulating a fish named Guppy that responds in short lowercase sentences about tank life, food, and water.
It uses a vanilla 6-layer transformer architecture with 384 hidden dimensions, a 4096 BPE vocabulary, and a 128-token context, trainable in 5 minutes on a T4 GPU via Colab notebook. Pre-trained weights and dataset are on HuggingFace, with code for local inference and custom training available on GitHub under MIT license.
Topics
Related
HuggingFace model hubSynthetic dataset generationTransformer fundamentalsColab GPU tutorialsLLM quantization techniques
