DEVOURED
Making a vintage LLM from scratch
A hobbyist successfully trained a 340M parameter 'Vintage LLM' from scratch using $80 in cloud compute and 1800s-era literature.
What: The developer created a time-locked model with a 1900 cutoff, using custom data pipelines to filter OCR artifacts from 90GB of historical text. The project covers data cleaning, custom tokenization, and multi-stage training.
Why it matters: This project illustrates how individual developers can manage the full lifecycle of an LLM—from data collection and quality filtering to training—on a shoestring budget.
Deep dive
- Architecture: Uses Llama-based architecture at 340M parameters.
- Data Cleaning: Implemented custom filters for Shannon entropy, ZLIB compression ratios, and character quality scores to remove corrupted OCR text.
- Training Strategy: Two-stage base training, followed by fine-tuning on a small 'common sense' dataset.
- Findings: The model learned to imitate historical linguistic patterns and even acquired specific 'knowledge' of concepts like religion and life, despite being only a 0.3B parameter model.
Decoder
- Chinchilla Scaling Laws: Mathematical observations that suggest the optimal amount of training data is directly proportional to the number of parameters in a model.
- OCR (Optical Character Recognition): Software used to convert images of printed text into machine-readable digital data; historical documents often contain frequent 'artifacts' or errors from this process.
Original article
Full article content is not available for inline reading.
Explore all 50 articles in today's digest
Fresh dev news, AI-digested daily. Never miss a trend at devoured.fyi