NanoChat

NanoChat is an open-source, full-stack training and inference pipeline for building a ChatGPT-style large language model from scratch, created by Andrej Karpathy. Written in approximately 8,000 lines of Python (PyTorch) and Rust, it covers the entire LLM lifecycle: tokenization, pretraining on FineWeb, supervised fine-tuning, reinforcement learning, and inference with a ChatGPT-like web UI. A basic model can be trained for as little as $100 (~4 hours on an 8×H100 GPU node), making LLM development accessible to researchers, developers, and learners. It also maintains a community leaderboard for a 'GPT-2 speedrun' to gamify and incentivize progress.