LLMs-from-scratch: Build an LLM in PyTorch

Key takeaways#

• LLMs-from-scratch is the official code companion for Sebastian Raschka's book Build a Large Language Model (From Scratch).
• The repository teaches a GPT-style language model pipeline in PyTorch without leaning on external LLM frameworks for the core implementation.
• It is best for developers who already know Python and want to understand tokenization, attention, pretraining, and finetuning by writing the pieces themselves.
• The main chapter code is designed to run on conventional laptops; a GPU is helpful but not required for the core learning path.

What this resource is#

LLMs-from-scratch is a hands-on educational repository for builders who want to understand large language models from the inside out. Instead of starting with a hosted API or a high-level training framework, the project walks through the pieces behind a small GPT-like model: text preparation, tokenization, dataloaders, attention, transformer blocks, pretraining, evaluation, and instruction finetuning.

The repository is the official code base for Sebastian Raschka's book Build a Large Language Model (From Scratch). That matters because the code is structured as a learning sequence, not just a research dump. Each chapter maps to a specific part of the LLM development workflow, so readers can move from a blank Python file toward a working model while seeing the tradeoffs behind each component.

Who should use it#

Use this resource if you are a Python developer, ML engineer, or technical founder who wants a durable mental model for how LLMs work. It is especially useful before you start fine-tuning larger open models or building infrastructure around agents. The repo does not try to make you a frontier model lab. It gives you the smaller, inspectable version of the same workflow.

It is less useful if you only want a copy-paste API tutorial. The value is in implementing pieces yourself and reading the code closely. The project assumes strong Python experience and benefits from basic familiarity with neural networks or PyTorch.

What you will learn#

The learning path covers text data handling, tokenization, batching, causal attention, transformer layers, GPT-style model construction, training loops, loading pretrained weights, classification finetuning, instruction finetuning, and LoRA-style parameter-efficient finetuning. That spread makes it a good bridge between practical engineering and model internals.

Getting started#

Clone the repository with:

git clone --depth 1 https://github.com/rasbt/LLMs-from-scratch.git

Then follow the setup notes in the repository's setup directory. If you are new to PyTorch, start with the appendix material or the author's linked PyTorch introduction before jumping into the model chapters.

Practical notes#

The repository had about 94.6k GitHub stars and 14.5k forks when this listing was verified on May 14, 2026. Its main languages are Jupyter Notebook and Python. The main code path is intended to be runnable on normal laptops, though training larger variants or experimenting beyond the book examples can benefit from GPU access.