Micrograd Autograd Engine Reference

Key takeaways#

• Micrograd is Andrej Karpathy’s tiny educational autograd engine and neural-network library.
• The repository implements reverse-mode automatic differentiation over scalar values with a compact PyTorch-like API.
• It is best used as a learning resource for understanding backpropagation, not as a production machine-learning framework.
• The official README includes a scalar example, a neural-network demo notebook, tests, and visualization helpers.

What this resource is#

Micrograd is a practical source-code reading resource for developers who want to understand how automatic differentiation works. The official repository describes it as a tiny autograd engine and a small neural-network library on top of it. The project is intentionally small: the README says the engine is about 100 lines of code and the neural-network layer is about 50 lines of code. That constraint is the point. Instead of hiding backpropagation behind a large framework, micrograd makes the computation graph, values, gradients, and backward pass easy to inspect.

How micrograd works#

Micrograd builds a dynamic directed acyclic graph during the forward pass. Each scalar Value records the data, the operation that created it, and the local backward function needed to move gradients back through the graph. When a user calls backward(), the engine walks the graph in reverse topological order and accumulates gradients. The implementation operates over scalar values rather than tensors, so a neuron is decomposed into individual adds, multiplies, powers, and nonlinearities.

That scalar-only design is not meant for speed. It is meant for clarity. Developers can trace every operation, inspect the graph, and understand why the chain rule produces each gradient. The README example computes a scalar expression, calls g.backward(), and prints both the forward value and gradients with respect to the inputs.

What you can learn from it#

Micrograd is useful for learning reverse-mode autodiff, computation graphs, neural-network modules, and the relationship between a tiny educational engine and larger systems such as PyTorch. The repository includes micrograd/, tests, a demo notebook, a graph-tracing notebook, and sample images. The demo notebook trains a small two-layer neural network on a binary classification task using a simple max-margin loss and stochastic gradient descent.

A good study path is to read engine.py first, then inspect the neural-network layer, then run the scalar README example, then run the demo notebook. After that, compare the same model idea with a tensor framework. The contrast shows why tensorization matters for performance while preserving the core backpropagation idea.

When to use it#

Use micrograd when teaching autodiff, preparing for framework internals work, or debugging your mental model of backpropagation. It is also useful as a compact codebase for explaining why modern AI libraries can compose arbitrary operations and still compute gradients automatically.

Do not treat it as a drop-in replacement for PyTorch, JAX, or TensorFlow. It is scalar-valued, educational, and intentionally tiny. For production training, use a real tensor library with vectorized kernels, hardware acceleration, serialization, distributed training support, and mature ecosystem tooling.

Verification notes#

This resource was created from the official public repository at https://github.com/karpathy/micrograd. Repository metadata can change quickly, so builders should check the README, license, demo notebooks, package setup, and tests before using the material in a course, article, or internal training workflow.