Ask an AI a question and watch the words arrive, one after another. It feels like the machine is looking something up. It is not. At every step, the model is placing a bet. Given everything written so far, it assigns a probability to each possible next token, picks one, and moves on. The paragraph you finally read is a chain of weighted guesses, and the quality of the whole answer depends on how well those guesses were calibrated. It is also why asking the same question twice can give you two different answers.
We are used to computers being certain. A calculator never hesitates over 7 times 8. But the systems now writing our emails, reading our medical scans and ranking our search results live inside uncertainty, and the mathematics of living inside uncertainty is probability.
That is why a Stanford course called Probability for AI exists, and why I am proud to be a small part of it. I am a section leader in the course, and this post is what I have learned so far.
The Course in One Paragraph
Probability for AI is a free online course created by Chris Piech, an Assistant Professor (Teaching) of Computer Science at Stanford and a Faculty Affiliate of Stanford HAI, the university's institute for human-centered AI (his Stanford profile). Piech describes it as a course that teaches you "the math language that AI speaks." Stanford HAI describes it as a six-week program for anyone who wants to understand how AI works, not just how to use it, with no prerequisites and no experience needed.
"Not just how to use it" is the whole point. Anyone can call an API. Understanding why an answer came out the way it did, why it is sometimes confidently wrong, and how much to trust it, requires the language of probability. And although the course welcomes beginners, it does not stay shallow. In my section, we go deep into how probability actually does its work inside AI.
How I Ended Up in the Room
I was selected as a section leader out of more than 2,700 participants from over 70 countries. A section is a small group where students work through the week's ideas together and get to ask questions. Mine is made up of working professionals and technologists from around the world.
I do it alongside a full-time job and all the usual weight of daily responsibilities, and it asks for real effort every single week. But I am enjoying this journey seriously, and being trusted to teach in a Stanford course is an honor I do not take lightly.
One Question That Shows Why Probability Matters
Try this one. It is the kind of question the course trains you to answer correctly.
A medical screening test catches 95 percent of people who have a certain condition. It also gives a false alarm for 5 percent of healthy people. The condition affects 1 percent of the population. You take the test and it comes back positive. What is the chance you actually have the condition?
Most people say about 95 percent. The real answer is about 16 percent.
Imagine 10,000 people. Of them, 100 have the condition and 95 of those test positive. Of the 9,900 healthy people, 5 percent, which is 495, also test positive. So among the 590 people with a positive result, only 95 are truly sick. That is roughly one in six.
Nothing in that calculation is advanced. Yet our intuition gets it badly wrong, and the same mistake hides inside AI systems that flag fraud, detect disease or screen applicants. If you build or use such systems, this is basic literacy, not a luxury.
Week 1: Same Prompt, Twice
Week 1 opened with something every AI user has quietly noticed: ask the same question twice and you may get two different answers. Why?
In our section, we explored it with a toy language model small enough to hold in your head. Give it the prompt "A sleepy robot dreams about" and it has exactly five possible outputs: electric sheep (44 percent), counting stars (20), sample spaces (14), counting outcomes (13) and robot puppies (9). Each call picks one at random, independently of the last.
So what is the chance that two calls give two different outputs? Three simple rules are enough: NOT, OR and AND. The chance that both calls give the same output is the sum of each output's probability squared, which comes to 0.2782. The chance of two different outputs is 1 minus that, about 72 percent. Even a model with a clear favorite disagrees with itself most of the time.
Then came a question that catches almost everyone. "Robot puppies" appears with a 9 percent chance on each call. How many calls until you are more likely than not to see it at least once? The tempting answer adds the chances: six calls, 6 times 9 percent, is 54 percent. But those chances overlap, so adding them overcounts. The right way is to ask for the chance of never seeing it. That is 0.91 on each call, so 0.91 multiplied by itself n times. After six calls the real chance is only 43 percent. It takes eight calls to pass 50 percent.
Finally, we met the dial that every AI tool hides behind a slider: temperature. It raises each probability to the power 1/T and rescales them so they add up to 1. Here it is in a few lines of Python, applied to the first-word chances of our toy model:
import numpy as np
p = np.array([0.44, 0.33, 0.14, 0.09]) # first word: electric, counting, sample, robot
def with_temperature(p, T):
q = p ** (1 / T) # raise each chance to the power 1/T
return q / q.sum() # rescale so they add up to 1
for T in (0.5, 1, 2):
q = with_temperature(p, T)
print(T, q.round(4), round((q ** 2).sum(), 4))
# 0.5 [0.5863 0.3298 0.0594 0.0245] 0.4567
# 1 [0.44 0.33 0.14 0.09 ] 0.3302
# 2 [0.3469 0.3005 0.1957 0.1569] 0.2736
The last number on each line is the chance that two calls start with the same word. At T = 0.5, "electric" climbs from 44 to about 59 percent, "robot" shrinks from 9 to about 2.5 percent, and two calls match about 46 percent of the time. At T = 2, the chances even out and two calls match only about 27 percent of the time. In real models, the same thing is done by dividing the logits by T before the softmax, which is mathematically the same operation.
The randomness setting in an AI tool is not magic. It is probability, rescaled.
I turned this section into a detailed solutions PDF with every step worked out, and a Manim animation so that students could revisit the ideas at their own pace, long after the live session ended.
Week 2: The Palpatine Poll
In Week 2, our section centered on a problem called the Palpatine Poll, and it was my favorite kind of problem: simple mathematics wrapped around a clever idea.
The question is how you measure something people will not say out loud. If you ask directly, many will hide the truth. The trick is called a list experiment. Split people into two groups. Show the first group a list of a few harmless statements and ask only how many they agree with, never which ones. Show the second group the same list with one sensitive statement added, and ask the same question. Nobody reveals a single individual answer, yet the second group's average count comes out higher, and that gap estimates how many people agree with the sensitive statement.
Underneath this trick sit two of the most useful ideas in probability. A single yes or no answer is a Bernoulli random variable, the simplest random event there is. Add up many of them and you get a binomial distribution, which tells you not only the expected count but also how much it is likely to wobble from one group of people to the next. That wobble is what separates a trustworthy estimate from a lucky one.
Where This Shows Up in Your ML Work
Every time a team reports that a model is "90 percent accurate," they are reporting a binomial experiment: run the model on n test examples and count how many it gets right. Treat each test example as an independent trial with a 90 percent chance of success, and you can compute exactly how much the reported number will wobble:
from scipy.stats import binom
p = 0.9 # the model's true accuracy
# Chance the observed accuracy lands between 84% and 96% on 100 test examples
print(binom.cdf(96, 100, p) - binom.cdf(83, 100, p)) # about 0.97
# Chance it lands within 0.6 points of 90% on 10,000 test examples
print(binom.cdf(9060, 10_000, p) - binom.cdf(8939, 10_000, p)) # about 0.96
On 100 examples, the number correct will land anywhere from about 84 to 96 in roughly 97 out of 100 runs. On 10,000 examples, the same model lands within about 0.6 of a percentage point of 90 percent, about 95 percent of the time.
Same model, same skill, wildly different certainty. Whether a headline number deserves your trust depends on a calculation most people never see.
Making the Hard Parts Visible
After each week, I do something extra. I write a detailed solutions PDF, and I build a Manim animation that shows the idea moving, not just written down. After Weeks 1 and 2, I shared both with my students and posted them on the section leaders' forum so that every section leader could use them too.
The reaction was the best surprise of the whole experience. Prof. Chris Piech replied to my post, calling a Manim video for Week 2 a real gift to students and to every section leader in the course, and a fellow section leader added kind words as well.
A formula on a slide asks you to trust it. An animation lets you watch the reason. When a binomial distribution widens and narrows as you change the number of trials, it stops being a symbol and becomes something you have seen. That has been true in my own classroom as well, and it is why I animate everything I can.
If you want to explore the material yourself:
- Detailed PDFs for Week 1 and Week 2: open the folder
- The animation playlist: Stanford Probability for AI on YouTube
What I Am Taking Away
Teaching this course has changed how I read AI headlines. When I see that a model is "accurate," I ask how many examples it was tested on. When a system is described as "confident," I ask whether that confidence is calibrated. When someone says an AI "knows" something, I remember that it placed a bet.
That habit of asking is worth more than any single formula, and you do not need to be a mathematician to build it. You need a good teacher, a few well chosen examples, and the patience to look at an idea until it moves. If you want to see where this course lives, it is at pai.stanford.edu.
About the author
I'm Sohaib Hasan. I have 9+ years of teaching experience in advanced mathematics and statistics, and I turn the ideas I teach into Manim animations, the kind that Prof. Chris Piech and fellow section leaders of Stanford's Probability for AI have appreciated.
I share these animations on my YouTube channel, PlotLab Algorithms, where I teach data science, machine learning, cryptography, math, DSA and statistics visually, in the style made popular by 3Blue1Brown. Shorter clips live on Facebook (80K+ followers) and Instagram (15K+ followers).
Want an animation like this for your own work? If you have a project, a research article, a thesis defense, a YouTube video, or any idea that is hard to explain with slides alone, I can turn it into a clear, professional animation. Reach out through my Work With Me page and tell me what you are working on. If you would rather learn to build animations yourself, start with my Manim CE Beginner's Handbook. And I'd like to hear what you think of this post, drop a comment.
Top comments (0)