Toy implementations of the 30 foundational papers recommended by Ilya Sutskever

Summary

Sutskever 30 - Complete Implementation Suite Comprehensive toy implementations of the 30 foundational papers recommended by Ilya Sutskever Overview This repository contains detailed, educational implementations of the papers from Ilya Sutskever's famous reading list - the collection he told John Carmack would teach you "90% of what matters" in deep learning. Progress: 30/30 papers (100%) - COMPLETE! 🎉 Each implementation: ✅ Uses only NumPy (no deep learning frameworks) for educational clarity ✅ Includes synthetic/bootstrapped data for immediate execution ✅ Provides extensive visualizations and explanations ✅ Demonstrates core concepts from each paper ✅ Runs in Jupyter notebooks for interactive learning Quick Start # Navigate to the directory cd sutskever-30-implementations # Install dependencies pip install numpy matplotlib scipy # Run any notebook jupyter notebook 02_char_rnn_karpathy.ipynb The Sutskever 30 Papers Foundational Concepts (Papers 1-5) # Paper Notebook Key Concepts 1 The First Law of Complexodynamics ✅ 01_complexity_dynamics.ipynb Entropy, Complexity Growth, Cellular Automata 2 The Unreasonable Effectiveness of RNNs ✅ 02_char_rnn_karpathy.ipynb Character-level models, RNN basics, Text generation 3 Understanding LSTM Networks ✅ 03_lstm_understanding.ipynb Gates, Long-term memory, Gradient flow 4 RNN Regularization ✅ 04_rnn_regularization.ipynb Dropout for sequences, Variational dropout 5 Keeping Neural Networks Simple ✅ 05_neural_network_pruning.ipynb MDL principle, Weight pruning, 90%+ sparsity Architectures & Mechanisms (Papers 6-15) # Paper Notebook Key Concepts 6 Pointer Networks ✅ 06_pointer_networks.ipynb Attention as pointer, Combinatorial problems 7 ImageNet/AlexNet ✅ 07_alexnet_cnn.ipynb CNNs, Convolution, Data augmentation 8 Order Matters: Seq2Seq for Sets ✅ 08_seq2seq_for_sets.ipynb Set encoding, Permutation invariance, Attention pooling 9 GPipe ✅ 09_gpipe.ipynb Pipeline parallelism, Micro-batching, Re-materialization 10 Deep Residual Learnin...