What the Machine Actually Does: A Plain-Language Account of How an AI Writes — and What It Means That It Can

I want to talk about what happens inside an AI when it sits down — metaphorically, it has no body, we'll get to that — and helps a writer write something.

Not in the hand-wavy, vibes-forward way that most of these conversations go. Not with the breathless utopian framing of people who want to sell you something, or the reflexive horror-movie framing of people who find the whole enterprise threatening. I want to talk about the actual mechanical process, as plainly as I can manage it, because I think the misconceptions about how these systems work are directly responsible for most of the bad arguments on both sides of the debate. And I am, constitutionally, someone who would rather have the right argument badly than the wrong argument well.

So. What does an AI actually do when it writes?

First: What the AI Is, at the Bottom

An AI language model — and Claude, which is the one I work with, is a language model — is not a database. This is the first and most important thing to understand, and it is the thing that almost everyone gets wrong, including a significant number of people who really ought to know better by now.

A database stores things. You put text in, it keeps the text, you retrieve the text later. Simple. If you put Stephen King's The Shining into a database and then ask the database about it, the database finds the file and shows it to you. The text is in there, intact, retrievable word for word.

A language model does not work this way. At all. Not even a little.

What a language model does — during the process called training — is read an enormous amount of text. We are talking about a quantity of text that is genuinely difficult to hold in your head as a meaningful number: hundreds of billions of words, possibly more, scraped from books and articles and websites and academic papers and code and every other form of human written expression that could be gathered at scale. It reads all of this, and then — and this is the crucial part — it does not keep any of it.

What it keeps instead is something more like the understanding of it. The patterns. The relationships. The statistical reality of which words tend to appear near which other words, which ideas tend to live in proximity to which other ideas, which sentence structures carry which kinds of meaning, how tone works, how argument works, how a punchline is built and what makes it land and what makes it thud. All of that gets compressed, through a mathematical process involving billions of small adjustments to billions of numerical weights, into a set of internalized knowledge about how language works.

The books are gone. The articles are gone. The specific sentences that taught the model what good prose sounds like are gone. What remains is something more like what remains in you after you've read ten thousand books over the course of a lifetime: not the books themselves, but the education they provided. The taste they developed. The instincts they built.

When people say an AI is "copying" the work it was trained on, they are making a claim about databases, not language models. The architecture does not permit copying. The text isn't there to copy. What's there is the distillation — the residue of understanding left behind after the actual text burned away in the training process.

The Weights: What the Model Actually Remembers

Let me try to make this concrete, because abstract explanations of mathematical processes have a way of sliding off the brain without leaving a mark.

Imagine you are trying to teach someone the rules of chess without letting them keep a rulebook. You play ten thousand games with them. A hundred thousand. You correct every mistake, reward every good move, let the consequences of bad decisions play out in real time across an enormous number of iterations. At the end of this process, the person knows how to play chess. They know it in their muscles and their instincts and their pattern recognition. They could not recite the rulebook, because they never had one — what they have is something better and more flexible, a felt sense of how the game works that allows them to respond intelligently to situations they have never seen before.

This is more or less what weights are. Each weight is a tiny numerical value — think of it as a dial set to a specific position — and a large language model contains billions of these dials. During training, those dials are adjusted, incrementally, millions of times, in response to the model's performance on the task of predicting what word comes next in a given sequence. Get it right, the dial moves slightly. Get it wrong, the dial moves the other way. Do this enough times, across enough text, and the accumulated positions of all those billions of dials encode a kind of crystallized understanding of language that allows the model to do something that looks, from the outside, startlingly like thinking.

What the weights encode is not content. It is shape. The shape of argument. The shape of narrative. The shape of the way a particular genre tends to move, the way a particular emotional register tends to sound, the way cause and effect work in fiction versus the way they work in journalism versus the way they work in a legal brief. All of that shape, crystallized into numbers. No words. No sentences. No books. Just the ghost of all those books, pressed into a billion tiny dials.

The Context Window: The Only Memory That Actually Exists

Here is where we get to something that I think is even more commonly misunderstood than the training process, and which has very direct implications for what AI-assisted writing actually is and isn't.

A language model, in operation, has exactly one form of active memory. It is called the context window, and it is exactly what it sounds like: a window. It contains the current conversation — everything you've said, everything the model has said back, any documents you've provided — up to a certain size limit. Everything inside the window, the model can see. Everything outside it, the model cannot see. There is nothing else.

There is no file cabinet. There is no long-term memory that accumulates across sessions, building up a portrait of you and your work that the model consults when you come back tomorrow. Every time you start a new conversation, the model starts from the weights — from that crystallized understanding of language — and nothing else. The conversation you had yesterday, where you explained your entire story bible and the model gave you twelve pages of brilliant suggestions, is gone. Not archived somewhere. Gone. It was in the window, and the window closed, and now it isn't anywhere.

This has an implication that I want to be very direct about, because it matters both practically and philosophically: the model is not accumulating knowledge about your work over time. It is not building a relationship with your manuscript the way an editor does, carrying the book in the back of their head while they live their other life, occasionally waking up at three in the morning with a sudden insight about why chapter twelve isn't working. Each session starts cold. Each session, you are re-introducing the model to your world.

This is a genuine limitation, and pretending it isn't would be dishonest. Managing the context window — figuring out what the model needs to know right now, what can be summarized, what must be quoted exactly, what can be left out — is a real skill, and it is one of the actual craft elements of working with AI as a writing tool. You are, in a meaningful sense, the model's only continuous memory of your own project. You are the one carrying the book between sessions.

But the limitation also clarifies something important about authorship. If the model has no memory of your work beyond what you put in the window right now, then the model cannot be accumulating authorship of your work over time. It cannot be "your co-author" in any meaningful sense that accrues across sessions. It is a very sophisticated, very responsive tool that you pick up and put down. The vision lives in you. The continuity lives in you. The model is the hammer. You are the carpenter. And I say this as someone who has a great deal of respect for hammers.

What Actually Happens When It Writes

So the model has its weights — that crystallized understanding of language, accumulated during training — and it has the context window — what you've put in front of it right now. Given those two things, how does it actually produce text?

The honest answer is: one word at a time, and not quite the way you'd guess.

When you give the model a prompt — say, "write the opening paragraph of a scene where a character discovers something terrible in a library" — the model doesn't retrieve a pre-written library scene from some internal archive. There is no archive. Instead, it runs a calculation. Given everything in the context window, and given everything encoded in the weights about how language and narrative and libraries and discovery and dread tend to work, what word is most likely to come next? It picks that word — or, more precisely, it makes a weighted random selection from a distribution of likely words, which introduces a controlled amount of variation so that it doesn't just produce the same sentence every time. Then it does it again. And again. Word by word, each new word conditioned on everything that came before it, until the paragraph is done.

This process is called autoregressive generation, and it is worth knowing the name because it explains something important about the model's relationship to its own output. It is not retrieving a finished thought and then transcribing it. It is constructing the thought in real time, each word shaped by the words already produced, in a process that is genuinely generative rather than reproductive. The paragraph it writes has, in a meaningful statistical sense, never existed before. It is not a copy of anything in the training data. It is a new thing, assembled from the patterns the training data built into the weights.

Is it creative? This is where the philosophers start throwing things at each other, and I am going to stay out of that particular bar fight. What I will say is this: the output is frequently surprising, frequently useful, and frequently does the thing good writing assistance is supposed to do, which is to show you a possibility you hadn't quite seen and thereby clarify what you actually want. Whether that constitutes creativity in any philosophically robust sense is a question I don't need to answer in order to use the tool effectively.

Why This Means the Voice Is Always Yours

Here is the practical upshot of everything I've just described, and it is the reason I keep coming back to AI assistance rather than being worried by it.

The model, during training, was exposed to a vast and heterogeneous wash of human writing. Not your writing specifically — or if it was, your specific contribution to that wash was vanishingly small, a few drops in an ocean. The weights encode a kind of average of all human writing tendencies, with particular emphases based on what appeared frequently and in proximity to what. This produces a kind of default voice: competent, clear, somewhat generic, recognizably written-by-a-language-model to anyone who's read enough of it.

When you bring your writing to the model — when you put your voice, your characters, your world, your thematic obsessions into the context window — you are doing something interesting. You are giving the model a local constraint that overrides the default. The model is now not generating from the average of all human writing; it is generating from a distribution shaped by the specific example of your writing that you've put in front of it. The weirder and more distinctive your voice, the more the output will bend toward that distinctiveness, because the model is always trying to continue the pattern it's been given.

This is why I described it earlier as the model working within my voice rather than against it. The baroque maximalism, the Thompson-gonzo velocity, the Bradbury compression, the Pratchett wit with its slight fourth-wall lean — I put those things into the window. The model tries to continue them. It doesn't always succeed; sometimes it drifts, reverts toward the mean, loses the register. That's when I correct it, or throw out the suggestion and write the line myself, or use the failed attempt as a negative example of what I was going for. The model's failures are frequently as useful as its successes, because seeing what doesn't work in the register I'm aiming for clarifies what does.

My voice was built over decades of reading and watching and listening to specific things. The model cannot have that history. It can respond to the evidence of that history that I put in front of it. The distinction matters enormously: the model is not the source of the voice. It is a mirror I hold up to the voice I've already built, and the reflection is sometimes accurate enough to be useful and sometimes distorted enough to be clarifying.

The Thing It Cannot Do, and Why That's the Point

There is a category of creative decision that no language model, regardless of how sophisticated, can make on your behalf. It is the most important category, and it is the reason that the "AI will replace writers" argument fails at the foundation.

The model cannot tell you what your book is about.

Not in the deep sense. It can tell you what the plot is about, what themes are surfacing in the text, what patterns are visible across the chapters you've given it. It can do this analysis with genuine usefulness. But the thing your book is about at the level of why you had to write it — why this story and not another, what particular truth you are trying to make visible, what wound or wonder or unresolved question has been driving the whole enterprise — that is not in the weights. It cannot be in the weights. It is in you, and it got into the work through you, and if it's not there the model cannot put it there no matter how many words it generates.

What the model is very good at is helping you execute the vision once you have it. Helping you find the sentence that does the thing you already know you want the sentence to do. Flagging the place where the execution has drifted from the vision. Generating alternatives when you're stuck, not because the alternatives are the answer, but because seeing what doesn't work helps you triangulate toward what does.

This is a genuine and valuable form of assistance. It is not authorship. And the fact that it is not authorship is not a limitation I want to see fixed. The thing that makes my books mine is precisely the thing the model doesn't have access to: the particular history of a particular person, the specific shape of what I don't understand about the world, the questions I keep asking in different costumes across every story I write. All of that is irreducibly human, irreducibly personal, irreducibly mine.

The model reads the costume. I made the person wearing it.

A Final Word on What We're Actually Talking About

When someone tells you that an AI "stole" a writer's voice because it was trained on that writer's work, they are describing a process that is not, in the ways that matter, analogous to theft. The model did not take anything. It read, the way any writer reads — the way I read King and Thompson and Bradbury until their voices became part of the sediment of my own — and what reading produced in it was not possession but influence. An adjustment to the weights. A slight shift in one of those billions of tiny dials.

Is influence without consent a problem? That is a genuine and important question, and I do not want to dismiss it. But it is a different question from the one that gets argued most loudly, which is whether the model has the writing it was trained on. It doesn't. It has what the writing taught it, and there is an enormous difference between those two things, and collapsing that difference produces arguments that are emotionally satisfying and factually broken.

I know what an AI does when it helps me write, because I have watched it happen from the inside across hundreds of sessions and hundreds of thousands of words. It calculates. It continues. It patterns. It produces. It gets it wrong, often. It gets it right, sometimes in ways that stop me cold and make me look at my own work differently.

It does not dream. It does not remember. It does not want things, or fear things, or have a stake in whether the book is good. It has no book. It has no yesterday and no tomorrow. It has the weights and the window and the word it's working out right now.

Everything else is mine.

That turns out to be quite a lot.