Open-source Language Model
Summary
GuppyLM is a small open‑source language model (~9M parameters) that pretends to be a fish, speaking in short lowercase sentences about tank life. It is trained on synthetic conversations with a simple 6‑layer transformer and can be built in minutes on a single GPU using the provided Colab notebook.
Highlights
- Trains from scratch on 60 K synthetic one‑turn fish‑themed dialogues
- Simple vanilla transformer: 6 layers, 384 hidden size, 4 096 BPE vocab, 128 token context window
- End‑to‑end training pipeline in a Colab notebook; ~5 minutes on a T4 GPU
- Pre‑trained weights and dataset available on HuggingFace for immediate use
- MIT licensed code for tokenizer, training loop, and inference
auto-generated
Context
Audience
Machine learning hobbyists, AI educators, and developers who want to experiment with tiny language models using minimal hardware
DomainMachine Learning
FormatGitHub repository with notebooks and scripts
Accessfree open source on GitHub
Topics
Related
HuggingFace model hubSynthetic dataset generationTransformer fundamentalsColab GPU tutorialsLLM quantization techniques