About this channel
Andrej Karpathy's channel approaches language models from both the user's desk and the code that makes them work. Introductions to large language models appear beside detailed construction exercises…
Andrej Karpathy's channel approaches language models from both the user's desk and the code that makes them work. Introductions to large language models appear beside detailed construction exercises, with separate material on using the systems in everyday practice. That combination lets a viewer connect the experience of interacting with a model to questions about its underlying machinery.
The teaching format is explicit in the build-along series. Videos construct GPT in code, reproduce a GPT-2 model and work through a tokenizer. These are concrete components and implementations to study, rather than simply a discussion of what AI might become. A broad introduction or a deeper conceptual talk offers another route for viewers who want an overview before entering the code.
The makemore sequence provides a further progression through language modelling. Its subjects include multilayer perceptrons, activations, gradients, batch normalisation, backpropagation and a WaveNet implementation. The sequence places individual techniques within an ongoing building exercise. Alongside this technical material, an account of personal LLM use returns attention to what someone actually does with the models once they are available.
Programmers and learners who want to understand model construction are the natural audience for the coding videos. Viewers with less technical background can begin with the introductory talks and practical-use discussion, then decide how far into implementation they want to go. The channel's distinctive connection is between explanation and construction: a concept becomes something to follow in code, while a working model becomes something whose components can be examined more closely.
Latest episode
How I use LLMs
The example-driven, practical walkthrough of Large Language Models and their growing list of related features, as a new entry to my general audience series on LLMs. In this more practical followup, I…
The example-driven, practical walkthrough of Large Language Models and their growing list of related features, as a new entry to my general audience series on LLMs. In this more practical followup, I take you through the many ways I use LLMs in my own life.
Chapters
00:00:00 Intro into the growing LLM ecosystem
00:02:54 ChatGPT interaction under the hood
00:13:12 Basic LLM interactions examples
00:18:03 Be aware of the model you're using, pricing tiers
00:22:54 Thinking models and when to use them
00:31:00 Tool use: internet search
00:42:04 Tool use: deep research
00:50:57 File uploads, adding documents to context
00:59:00 Tool use: python interpreter, messiness of the ecosystem
01:04:35 ChatGPT Advanced Data Analysis, figures, plots
01:09:00 Claude Artifacts, apps, diagrams
01:14:02 Cursor: Composer, writing code
01:22:28 Audio (Speech) Input/Output
01:27:37 Advanced Voice Mode aka true audio inside the model
01:37:09 NotebookLM, podcast generation
01:40:20 Image input, OCR
01:47:02 Image output, DALL-E, Ideogram, etc.
01:49:14 Video input, point and talk on app
01:52:23 Video output, Sora, Veo 2, etc etc.
01:53:29 ChatGPT memory, custom instructions
01:58:38 Custom GPTs
02:06:30 Summary
Links
- Tiktokenizer https://tiktokenizer.vercel.app/
- OpenAI's ChatGPT https://chatgpt.com/
- Anthropic's Claude https://claude.ai/
- Google's Gemini https://gemini.google.com/
- xAI's Grok https://grok.com/
- Perplexity https://www.perplexity.ai/
- Google's NotebookLM https://notebooklm.google.com/
- Cursor https://www.cursor.com/
- Histories of Mysteries AI podcast on Spotify https://open.spotify.com/show/3K4LRyMCP44kBbiOziwJjb
- The visualization UI I was using in the video: https://excalidraw.com/
- The specific file of Excalidraw we built up: https://drive.google.com/file/d/1DN3LU3MbKI00udxoS-W5ckCHq99V0Uqs/view?usp=sharing
- Discord channel for Eureka Labs and this video: https://discord.gg/3zy8kqD9Cp
Educational Use Licensing
This video is freely available for educational and internal training purposes. Educators, students, schools, universities, nonprofit institutions, businesses, and individual learners may use this content freely for lessons, courses, internal training, and learning activities, provided they do not engage in commercial resale, redistribution, external commercial use, or modify content to misrepresent its intent.
YouTube Shorts
More episodes from this channel
Deep Dive into LLMs like ChatGPT
5.2.2025
This is a general audience deep dive into the Large Language Model (LLM) AI technology that powers ChatGPT and related products. It is covers the full training stack of how the models are developed,...
Let's reproduce GPT-2 (124M)
9.6.2024
We reproduce the GPT-2 (124M) from scratch. This video covers the whole process: First we build the GPT-2 network, then we optimize its training to be really fast, then we set up the training...
Let's build the GPT Tokenizer
20.2.2024
The Tokenizer is a necessary and pervasive component of Large Language Models (LLMs), where it translates between strings and tokens (text chunks). Tokenizers are a completely separate stage of the LLM pipeline: they have...
[1hr Talk] Intro to Large Language Models
23.11.2023
This is a 1 hour general-audience introduction to Large Language Models: the core technical component behind systems like ChatGPT, Claude, and Bard. What they are, where they are headed, comparisons and analogies to present-day...
Let's build GPT: from scratch, in code, spelled out.
17.1.2023
We build a Generatively Pretrained Transformer (GPT), following the paper "Attention is All You Need" and OpenAI's GPT-2 / GPT-3. We talk about connections to ChatGPT, which has taken the world by storm. We...
Building makemore Part 5: Building a WaveNet
21.11.2022
We take the 2-layer MLP from previous video and make it deeper with a tree-like structure, arriving at a convolutional neural network architecture similar to the WaveNet (2016) from DeepMind. In the WaveNet paper,...
Building makemore Part 4: Becoming a Backprop Ninja
11.10.2022
We take the 2-layer MLP (with BatchNorm) from the previous video and backpropagate through it manually without using PyTorch autograd's loss.backward(): through the cross entropy loss, 2nd linear layer, tanh, batchnorm, 1st linear layer,...
Building makemore Part 3: Activations & Gradients, BatchNorm
4.10.2022
We dive into some of the internals of MLPs with multiple layers and scrutinize the statistics of the forward pass activations, backward pass gradients, and some of the pitfalls when they are improperly scaled....
Building makemore Part 2: MLP
12.9.2022
We implement a multilayer perceptron (MLP) character-level language model. In this video we also introduce many basics of machine learning (e.g. model training, learning rate tuning, hyperparameters, evaluation, train/dev/test splits, under/overfitting, etc.). Links: -...
The spelled-out intro to language modeling: building makemore
7.9.2022
We implement a bigram character-level language model, which we will further complexify in followup videos into a modern Transformer language model, like GPT. In this video, the focus is on (1) introducing torch.Tensor and...
Stable diffusion dreams of steampunk brains
17.8.2022
Prompt: "ultrarealistic steam punk neural network machine in the shape of a brain, placed on a pedestal, covered with neurons made of gears. dramatic lighting. #unrealengine" Stable diffusion takes a noise vector as input...
The spelled-out intro to neural networks and backpropagation: building micrograd
16.8.2022
This is the most step-by-step spelled-out explanation of backpropagation and training of neural networks. It only assumes basic knowledge of Python and a vague recollection of calculus from high school. Links: - micrograd on...
Continue reading
You may also like
Innohub TV
Watch this content also on Innohub TV
We picked clips from Innohub TV that continue the same topic.
