Open a chatbot and ask it for a short story, one that did not exist a minute before you asked. Where did it come from? This chapter takes that apart, in three questions, taken in order.
First: what is a machine actually doing when it hands you a paragraph, or a picture, that did not exist before? Second: why does the same machine sometimes hand you a confident sentence that is simply false — and why doesn't a small fix mend that the way a patch mends an ordinary bug? Third: once making convincing content is this easy, what new questions of honesty travel with it?
One pair runs through every section ahead, and is worth naming now. A spam filter sorts your mail into a bucket decided long before your mail arrived. A chatbot writes you something that did not exist until you asked for it. Neither is a better version of the other. They are answering two different questions, and the gap between those questions is where this chapter lives.
Picture the machine that writes your poem as a clerk in a record room. Somewhere inside, in a labelled file, your poem is lying finished, and the clerk's whole job is to find it. Most of us carry some version of this picture — where else would the poem come from?
Test it. Ask for the poem twice. Two different poems come back. A clerk pulling a finished page from a file would bring the same page both times. So there is no file.
Then what is inside? Think of the man at the tea stall you cross every day after school. Thirty years at that counter, and every kind of talk has passed in front of him — results day, match day, bandh day. Has he memorised even one of those conversations? Not one. But say two sentences to him and he can tell where the third is going. Not because he has heard your third sentence before — because thirty years gave him a feel for what follows what.
A generative model holds exactly that kind of feel, learned from everything it trained on — not stored answers, but a statistical sense of what tends to follow what. Give it a prompt, and it draws one plausible reply from that sense, built fresh, word by word. Nothing is fetched from storage — every output is built at the moment you ask for it.
Weaker. The clerk in the record room. Send him for a page and he walks straight to the file where it has always sat, and brings back that very page, unchanged. Nothing about the file moves between one request and the next. The page existed before you asked; finding it, never making it, is the whole of his job.
Stronger. The tea-stall man. Put a question to him about a conversation he has never actually heard. He does not go quiet — he answers, fluently, out of thirty years of feel for how such talk usually runs. A generative model works the same way. It holds a learned distribution over everything it trained on, and at generation time it samples one plausible answer that fits your prompt. Run the same prompt again and the answer can come back different, and just as plausible — each pass is a fresh draw from that distribution, never a repeat visit to a file.
Close this page. In your own words — one sentence only — say how a generative model produces its output. Do not use the word database. Do not say look up. Now read your sentence back: does it say the model samples, draws, or builds its answer from a learned distribution? If it still sounds like something being found rather than made, write it again.
"Generative AI retrieves the answer from a large database."
It holds no database. There is no file anywhere holding your answer, ready to be found — that is the record-room picture, and it is the wrong one here. What a generative model holds is a learned distribution: a statistical sense, built from training data, of what tends to follow what. At generation time it samples one plausible output from that distribution, given your prompt — built fresh, not fetched from storage. That is exactly why the same prompt can come back with two different, equally plausible answers. Each run is a new draw, never a repeated visit to a file. A product built on top of a model can still attach a real retrieval step before generation, feeding it documents to draw from. But that is a step wrapped around the model — not the model itself keeping a database.
Generation does not happen in one jump. It happens piece by piece. At each step, the model draws its next piece from a distribution — a spread of options, each carrying its own weight, shaped by everything written so far.
Ask the tea-stall man the same question on two different afternoons. You may get two different, equally fluent replies — his feel for what comes next allows more than one honest answer. A generative model can do the same, for the same reason. Put the same prompt to it twice, and nothing about the model itself changes between the two askings — only the draw.
It is tempting to picture this as reaching into a dictionary and picking a word at random. Test that picture against what actually reads back: the draw is weighted, not random. Some next pieces are far more likely than others, and that weighting is exactly why the output reads as one connected passage, not as nonsense.
A spam filter does not work this way. Put the same email to it twice and the verdict does not move — its boundary stays fixed between askings. A generative model has no fixed boundary like that. It has a distribution, and every run is a fresh sample from it.
Now put a different question to a different kind of model. Instead of sorting an input into one of a fixed set of labels, a generative model learns the shape of the data itself. What would a plausible new instance of that data look like?
Give it a prompt, and it produces something new: a paragraph, an image, a tune that fits the shape it learned. The set of things it could produce was never fixed in advance — it stays open. A chatbot and an image generator both work this way. Neither is answering which label? Both are answering what would a new instance of this look like?
Generative AI is not one single thing. For this course, three output types matter most: text, image, and audio or video. The figure beside this paragraph draws the fuller map — five domains in all, counting video apart from audio and adding code as a fifth. One principle runs under every domain on that map: learn a distribution, sample from it. Your course takes a three-type slice of that same principle.
Why can't one model simply switch domains? The answer sits in the training data: a text tool learned from text, not from images, so it has nothing to draw an image from. The output type is not a setting you flip — it is baked into what the model learned from.
Text tools are large language models, predicting the next piece of text — you meet these next. Image tools are built one of two ways, which a later course covers. Audio and video tools follow the same one principle, applied to sound waves or video frames instead of words.
A large language model — an LLM — is a generative model trained on an enormous amount of text. It writes new text one piece at a time, predicting the next piece from the patterns it learned, given your prompt and everything it has already written. Chatbots, autocomplete tools, and text summarisers are all built this way.
You already know why an LLM is not a giant file of stored answers waiting in a record room — the same reason a generative model never is. Hold onto the one thing that matters most here: the loop predicts the next plausible piece. Nowhere in that loop is there a step that checks whether what it just wrote is true.
Bring back the tea-stall man. Put to him a conversation he has genuinely heard, and his feel for what usually comes next serves him well. Now put to him one he has never heard at all. He does not go quiet. He answers anyway, fluently and with confidence — because fluency is what thirty years gave him, not a memory of this particular exchange.
That is hallucination: a generative model producing a fluent, confident, false statement, for the same structural reason. It picks the next plausible piece, not the next true one, and nothing in its loop checks the second property. Plausible and true are independent qualities of a sentence — a model can nail the first and miss the second completely, in the same breath.
Does a bigger model outgrow this? It does not. As models grow larger, their errors get more fluent too, which makes a wrong answer harder to catch, not easier. The habit of checking matters more as the tools improve, never less.
Weaker. The spam filter's error. Every call it makes lands in one of four bins: correct spam, correct not-spam, or one of two mistakes — good mail marked spam, or spam let through. When the filter is wrong, the mistake lands in a known bin. Open the bins, count what landed where, and you know exactly how wrong the filter is.
Stronger. The LLM's error is not like that. A hallucination lands in no bin at all. The model has succeeded at its actual task — a plausible continuation — while failing at the task you assumed it was doing — stating a true fact. And that failure arrives wrapped in the same fluent, confident prose as a correct answer. No counter ticks. No bin fills. The error is structural, and the fluency is exactly what hides it.
Bring to mind the last confident AI answer you read. In one sentence, name how you would check it: which trusted source you would turn to, and which single claim in that answer you would test first. The habit you are practising is simple — give the most care to the answer that sounds the most certain.
"Hallucination is a temporary problem — future, larger models will fix it completely."
Hallucination comes from how generation works, not from a bug waiting to be patched. A model that draws its next piece as a weighted pick from a distribution can promise you something plausible — never something guaranteed true — no matter how large it grows. A bigger model gets more fluent, and its errors get more polished, not more truthful. Better models hallucinate less often. They do not stop hallucinating. That is a constraint you work around, not a defect that disappears on its own.
The strengths of generative AI and its limits trace back to the very same root. The model learns to produce likely-looking output, not correct output, and both lists follow from that one fact.
The strengths come from likely. It drafts fast. It offers real creative variety, and it adapts to your prompt. It works across text, image, audio, and video, which widens who can use it, and it works at a scale no single person could match.
The limits come from not correct. It hallucinates fluent falsehoods. It can amplify biases already sitting inside its training data. It holds no real understanding of what it is saying. It can reproduce copyrighted material, which raises intellectual-property risk no matter how it is credited. And training and running it costs real energy.
Hold both lists at once, and you get the honest picture. Generative AI is neither magic nor a toy.
"The limitations are temporary problems, not real constraints — they will disappear as models improve."
Hallucination and bias amplification are not bugs waiting for a patch. They come from how generation itself works. A model that picks the likely next piece cannot guarantee the true one, so a bigger model becomes more fluent, not more truthful. Its bias amplification grows right along with it, rather than shrinking. These are constraints you design around, not defects a future model removes just by getting bigger.
Creating with AI opens ethics questions that the verify habit alone does not cover, because now you are making content, not only reading it. The same model that writes you a fair summary of a real speech can write a confident, detailed speech that was never given at all. Same fluency, same polish — only the prompt changes.
That is what makes a deepfake possible: a generated image or video of a real person doing or saying something they never did, made convincingly enough to pass as real.
The test for whether this is wrong is not how good it looks. The test is consent. Making a realistic depiction of a real person without asking them first is harmful, however good the result looks.
One model can produce the helpful version and the harmful one with exactly the same fluency. The risk sits inside the mechanism itself, not only in the hands of a few bad actors.
ai-genai-ethics.ethics-synthetic-media
Ethical dimension
Synthetic-media misuse
The same generative model that makes harmless art can make a deepfake — a realistic image or video of a real person doing or saying something they never did. The helpful output and the harmful one come from the same mechanism. Only the prompt changes. That is why the risk is built into the capability itself, not added by a handful of bad actors.
Picture a convincing video of a real teacher appearing to say something they never said, shared online as if genuine. It is harmful no matter how good the video looks, because the test that matters is consent, not quality.
How do you keep the honest uses of generative media, and still stop the non-consensual fakes, when the tool itself cannot tell a harmless prompt from a harmful one?
"Synthetic-media harm only comes from a few bad actors deliberately misusing the tools."
The same model that writes a helpful summary can fabricate a convincing fake speech — only the prompt differs. The capacity to fabricate at scale, with the same fluency as an honest answer, is built into how generation works. That makes the risk structural, not personal. It needs no villain behind it, only the ordinary availability of a tool that cannot tell a helpful prompt from a harmful one on its own.
Here is a mistake worth admitting. I once thought disclosure only mattered when the AI-made result looked obviously fake — a rough image, a stiff sentence. Get a genuinely good result, I assumed, and the question quietly goes away. It does not.
Disclosure means clearly stating when a piece of text, image, audio, or video was made with AI help. It is required in three situations. When AI generated or substantially drafted the work. When the content depicts a real person. And when the content is presented as factual or documentary — where a reader would reasonably assume no AI was involved.
Skip disclosure when one of these applies, and you have deceived your reader, even if nothing in the content itself is false. Disclosure is a question of honesty, not of quality. An AI draft that reads exactly like something you would have written yourself still needs to be said out loud.
"If the AI-generated content is good quality, disclosure is unnecessary — what matters is the result, not the process."
Disclosure is about honesty, not about quality. A convincing image of a real person doing something they never did is more harmful the more convincing it looks, not less. The quality of the fake changes nothing about the consent question, or the deception question underneath it. The same rule holds for an AI draft that reads exactly like something you would have written yourself. If AI substantially made it, you say so.
One more question follows generative AI everywhere it creates: whose work made this possible? LLMs and image generators are trained on copyrighted text, images, and code, often without asking the people who made them. When a tool produces an image in a living artist's style, that artist's work shaped the result, and the artist gets no credit and no payment for it.
Three questions remain genuinely open, with no settled answer anywhere in the world as of now. Does training on copyrighted work break copyright law? Who owns the output it produces — the user who typed the prompt, the company that built the tool, or the people whose work trained the model? And does generating in someone's exact style violate their rights?
Courts and lawmakers everywhere are deciding these questions right now — there is no verdict yet to memorise.
ai-genai-ethics.ethics-copyright
Ethical dimension
Intellectual property & copyright
A generative model is trained on work made by real writers, artists, and musicians, often without their consent. When it produces output in someone's style, that person's work shaped the result, yet they receive no credit and no payment. The law has not yet settled who, if anyone, owns what the model produces.
An image tool generates a picture in the style of a living illustrator whose work sat inside its training data. The illustrator's years of labour shaped the output. Their name appears nowhere near it.
Does training on copyrighted work require the owner's permission? And who owns the result it produces — the user who typed the prompt, the company that built the tool, or the original creators whose work the model learned from?