Recommended Review Board

To evaluate this project, I recommend a panel consisting of Machine Learning Infrastructure Engineers and AI Educators. This project bridges the gap between deep-learning theory and accessible educational tooling, making it a subject of interest for those specializing in "TinyML," pedagogical AI design, and efficient architecture optimization.

Abstract

GuppyLM is an open-source, pedagogical language model project featuring an 8.7M parameter architecture designed to emulate the limited, environment-centric persona of a fish. The project emphasizes transparency and accessibility, providing a complete end-to-end pipeline—from synthetic dataset generation and tokenizer training to model architecture and inference—all executable in approximately five minutes on a single T4 GPU. The architecture utilizes a "vanilla" transformer design (LayerNorm, ReLU FFN, and learned embeddings) to demonstrate the fundamentals of LLM construction without the obfuscation of modern optimizations like GQA or RoPE, making it a viable tool for understanding model internals.

Summary: GuppyLM Technical Overview

• Model Architecture (8.7M Parameters): The model employs a streamlined, vanilla transformer design featuring 6 layers, 384 hidden dimensions, and 6 attention heads. It deliberately omits high-complexity features (e.g., SwiGLU, RoPE, GQA) to maintain code clarity and demonstrate core functional principles.
• Dataset Composition: Training relies on a 60,000-sample synthetic dataset (guppylm-60k-generic) covering 60 distinct, fish-appropriate topics (food, water, light, tank environment). The data uses template-based composition to ensure stylistic consistency in the "Guppy" persona.
• Inference Constraints: To maintain character stability, the model is limited to single-turn interactions. This design decision mitigates performance degradation observed in the 128-token context window during multi-turn testing.
• Pedagogical Goal: The project is designed to demystify LLM construction. By removing heavy system prompts and advanced optimization layers, the model forces the personality to reside directly within the weights, providing a clear mapping between raw input and specific output behavior.
• Training & Execution: The training loop utilizes standard PyTorch and HuggingFace libraries. The hardware requirements are intentionally low, allowing the entire training lifecycle to run in a Google Colab environment in under five minutes.
• Implementation Details: The codebase is modular, separating hyperparameter configuration, model architecture (model.py), data loading, and training loop logic to assist users in identifying how each component contributes to the final trained weight state.