Picture a photo restorer who has spent years fixing damaged photographs. Scratches, water stains, film grain, faded color. They've seen so much damage that they can look at a ruined photo and tell you exactly which marks are damage and which are the picture underneath.
Now hand them a frame of pure TV static with a note pinned to it: "This is a damaged photo of a corgi in a spacesuit. Please restore it." There's no corgi under there. There never was. But the restorer does what they always do: removes what looks like damage, a little at a time, and a corgi in a spacesuit appears. That is how most AI image generators work. (I used AI tools to help research and draft this post, and the illustrations were made with ChatGPT's image generator, which, as you'll see near the end, works a little differently.)
Step one: train a restorer by ruining good photos
You can't teach someone to remove damage without showing them damage. So training starts with a huge pile of good photos, each with a caption, the kind of image and alt-text pairs scraped from the web by the hundreds of millions or billions.
Take one photo. Add a random amount of noise to it: sometimes a light sprinkle, sometimes so much that the photo is almost gone. Show the model the noisy version, the caption, and how much noise was added. Then ask it one question: what exactly is the noise?
The model guesses. You already know the right answer, because you added the noise yourself. The difference between its guess and the truth is the error, and training nudges the model's weights to make that error smaller. Repeat this across the whole dataset, at every noise level, many times over.
Most diffusion models are trained this way to predict the noise, or a close mathematical cousin of it. They are never trained to "draw a corgi." They're trained to answer "which part of this is damage?" Drawing turns out to be a side effect.
Training: ruin a good photo by a random amount, then ask the model to point out exactly which part is the damage.
Step two: the work order is the prompt
A restorer with no instructions would still clean up static into something, but it would be a random something. The note pinned to the frame is what makes it a corgi.
Your prompt is that note. A text encoder (CLIP in the original Stable Diffusion; newer models such as Stable Diffusion 3 and FLUX add a larger language model encoder called T5) turns the words into a list of numbers that capture their meaning. While the model works, a mechanism called cross-attention lets every part of the image look back at the work order and ask which words matter for this spot. The patch that's becoming the helmet pays attention to "spacesuit." The patch becoming fur pays attention to "corgi."
This is also why the wording of a prompt matters so much. The model can only follow the note as well as the text encoder understood it.
Step three: restore in passes, coarse to fine
The restorer doesn't fix everything in one go. Each pass, they predict all the damage, remove part of it, and look again. A typical Stable Diffusion run does this 20 to 50 times. Each pass is called a step.
What changes from pass to pass is the interesting part. In the early passes the frame is almost all static, so the only decisions available are big ones: is the bright area at the top a sky? Is that dark blob in the middle a body? By the middle passes, the layout is fixed and shapes appear. The last passes are where fur, reflections, and eyelashes get decided.
That's why changing the prompt slightly can keep the same composition but change the details, and why tools that let you start from your own sketch work at all: they skip the early passes and hand the restorer a frame that already has the layout.
Early steps decide the layout. Late steps decide the eyelashes.
The seed is which static you were handed
Two restorers handed two different frames of static will find two different corgis. The static is random, and its exact pattern is set by a number called the seed.
Same prompt, same seed, same model, same settings, and you'll usually get the same image back. Change only the seed and you get a different corgi. That's why image tools let you "reroll," and why people share seeds when they want someone to reproduce a picture.
Guidance: what would you do without the note?
Here is a trick that makes prompts actually stick. At every step, the model makes two predictions. One with your work order, and one with a blank note. The difference between them is "what the prompt is adding." The sampler then pushes the image further in that direction than the model would on its own.
This is called classifier-free guidance, and the knob is the guidance scale. Stable Diffusion's common default is 7.5. Turn it down and the image drifts away from your prompt. Turn it way up and the restorer overcorrects: colors get oversaturated, contrast blows out, and the image looks "fried." It also doubles the work, since every step runs the model twice.
Negative prompts use the same machinery. Instead of a blank note, the second prediction uses a note that says "blurry, extra fingers," and the image is pushed away from it.
Guidance compares the restoration with the note and without it, then exaggerates the difference.
The restorer works on a thumbnail
Restoring every pixel of a big image, dozens of times, is slow. So models like Stable Diffusion don't work on pixels at all.
A separate model, called a variational autoencoder (VAE), first learns to shrink images into a compact sketch and blow them back up. In Stable Diffusion's original version, a 512×512 color image becomes a 64×64 grid with 4 channels: 48 times fewer numbers. All the noise and restoring happens on that small grid, called the latent. Only at the very end does the decoder enlarge the finished latent into a full image.
That's what "latent diffusion" means: the restorer works on a detailed thumbnail, and a very good enlarger prints the final photo.
All the restoring happens on a small latent. The decoder enlarges it at the end.
Putting it together
One image, start to finish.
Here's the whole run when you type a prompt and press generate:
- The text encoder turns your prompt into the work order.
- The seed produces a frame of random noise in the latent space.
- The denoiser predicts the noise, twice per step, with and without the prompt.
- Guidance exaggerates the difference, and the sampler removes part of the predicted noise.
- Steps 3 and 4 repeat, 20 to 50 times, coarse to fine.
- The VAE decoder enlarges the finished latent into pixels.
Why the restorer gets hands and text wrong
The model learns what damage looks like across millions of small patches, not what a hand is. Hands are small in most photos, bend in countless ways, and are often half hidden, so the statistics of "hand-shaped patch" are messy. The result was the famous six-finger problem.
Text was worse for a different reason: the original text encoders didn't see spelling well, so the restorer knew a sign should have writing on it but not which letters. Larger text encoders, bigger models, and better training data have improved both a lot, which is why newer generators can write words on signs.
Does it store the photos it trained on?
Mostly no. The model's weights hold patterns, not a library of images: a Stable Diffusion model file is a few gigabytes, while its training images would fill far more space than that. But researchers have shown that a model can reproduce a small number of training images almost exactly, usually ones duplicated many times in the dataset. "Learns patterns" is the normal case. "Never copies" is not a guarantee.
The new restorers
The workshop keeps upgrading. Newer models replace the original U-Net denoiser with a transformer, the same architecture behind chatbots, and many train with a method called flow matching that makes the path from static to image straighter, so fewer steps are needed. Distilled models can produce a usable image in as few as one to four steps.
And not every generator is a restorer. OpenAI has described GPT-4o's image generation, which made the illustrations in this post, as autoregressive: it builds the image piece by piece, more like writing a sentence one word at a time than cleaning up static. Different method, same goal.
The takeaway
An AI image generator doesn't paint. It was trained on one skill, spotting noise, and it creates by being handed pure noise and told what photo is supposedly hidden underneath. The prompt is the work order, the seed is the static, the steps are the passes, and guidance is how hard the restorer leans on your note.
Once you know that, the settings in every image tool stop being mysterious. What's the strangest thing you've ever watched one of these restorers "find" in the static?
Top comments (0)