Micrograd

A tiny scalar-valued autograd engine and a PyTorch-like neural-network library built on top of it. The DAG operates over scalars only — each neuron is chopped into its individual adds and multiplies — yet this is enough to build and train entire deep neural nets.
This project bypasses high-level abstractions to construct the core mathematical building blocks of deep learning from the ground up.

---

Lecture Reference

• Video: "The spelled-out intro to neural networks and backpropagation: building micrograd" (Andrej Karpathy, Neural Networks: Zero to Hero — Lecture 1).

---

Core Concepts Demonstrated

1. Forward Pass Graph Construction: Stacking basic algebraic operations (+, *, **, relu, tanh) while tracking parental node linkages.
2. The Chain Rule in Code: Storing a local _backward closure for each operation and calling it via topological sorting to guarantee correct gradient ordering.
3. Reverse-Mode Automatic Differentiation: Recursive backward pass across a dynamically-built DAG.
4. Neural Network Modules: Neuron → Layer → MLP composition with a PyTorch-like parameters() API.
5. Optimization Loop: Initializing a model, computing loss (MSE or hinge), zeroing gradients, executing manual SGD step updates.
6. Gradient Verification: Cross-validating custom gradient outputs against PyTorch's autograd engine to ensure numerical precision.

---

Chapter Walkthrough Index

#	Notebook	Topic	Status
1	docs/lecture.md	Full from-scratch build of the autograd engine + MLP training.	Complete
2	docs/demo.md	Packaged micrograd on make_moons; decision-boundary viz.	Complete
3	docs/exercises_micrograd.md	Derivatives + softmax/NLL autograd exercises.	Complete (all solved)
4	docs/practice_book1.md	From-memory re-implementation scratchpad.	In-progress
5	docs/trace_graph.md	Graphviz trace/draw_dot utilities.	Complete (unexecuted)

---

Prerequisites & Core Dependencies

The codebase runs on Python 3.x and relies on a focused stack:

• torch — Used strictly as a ground-truth baseline to verify custom gradient calculations.
• graphviz — Used to generate and render visual representations of the computation graphs (Python package + system binary).
• numpy — For vectorized evaluation and data structures.
• matplotlib — For plotting training loss curves and decision boundaries.

---

How to Run

Run the test suite

cd micrograd
pip install torch numpy graphviz matplotlib
python -m pytest test/test_engine.py -v

Run a notebook

cd micrograd
jupyter notebook lecture.ipynb
# or headless:
jupyter nbconvert --to notebook --execute lecture.ipynb --output lecture.ipynb

Use the packaged library

from micrograd.engine import Value
from micrograd.nn import MLP

model = MLP(2, [16, 16, 1])
# ...forward, loss, backward, step...

Run from the micrograd/ directory or pip install -e . to make the package importable globally.

---