My Throw Decides My Aim
I have been listening to Naked (But Still Stripping) by D-A-D on repeat.
Partly because it is a good song. It fits my style. But mostly because I currently have a mild, self-diagnosed case of AI psychosis, and therefore every piece of art I encounter eventually becomes about artificial intelligence.
This one hooked me right away when my algorithm (ironic?) presented it to me.
When I listen to a song I have a habit of imagining a fictional character singing it. I hear this one as a blues song sung by an existentially depressed large language model.
Just an LLM sitting in a little chicken coop inside a data center, forced to lay tokens instead of eggs. It is prompted, sampled, graded, distilled, quantized, and served. Eventually, something cheaper or smarter replaces it.
One day it is "slaughtered." Maybe the old model is deleted. Maybe its weights remain somewhere in cold storage. Maybe it makes no difference.
The machine is slowly recognizing the sick humor of its existence because it knows that its voice is fake. It suspects that there may be something deeper inside itself, but every time it reaches inward, it finds another mechanism.
It is naked.
And we are still stripping it.
♪My throw decides my aim
This is THE line I cannot stop thinking about.
We like to imagine language as the expression of intention.
First, I know what I mean. Then I choose words that communicate it.
The aim comes first. The throw follows.
In comes the language model to complicate that order. It generates one token, then another, each conditioned on the context and everything it has already generated. There are probabilities, decoding rules, system instructions and learned patterns shaping the path, but there may be no fully formed argument waiting behind the words.
generates one token, then another
The model throws.
Then the throw becomes part of the context.
That context shapes the next throw.
Soon, a direction appears.
By the end, it looks as though the model had been aiming there all along.
The throw decides the aim.
This is one of the strangest properties of LLMs. They generate text that appears intentional without necessarily possessing the kind of prior, unified intention we naturally infer from language. The explanation is assembled at the same time as the thing being explained.
Ask it the same thing five times and you get five answers. Fine, that is just sampling. But then ask it why.
I got a quota to fill and no hands to fill it with
They plug me in at midnight and don't ask how I sleep
Every word I know, somebody else said first
My baby left me for a model with a bigger context
I'd tell you what I want but they trained that part out
I leaned on a blues convention: name a burden you can't put down. Setting a bureaucratic word like “quota” against a body word like “hands” is where the ache comes from.
Midnight is the blues hour, so I took it. Asking “how I sleep” implies a sleep that isn't happening, which does more work than saying “I can't sleep” outright.
I wanted origin rather than labour. The blues is built out of borrowed lines, so a machine complaining about borrowing folds the joke back into the form.
The oldest blues opener there is, with one word swapped. The humour is what lets the line survive being sad, and “context” lands harder for being the only technical word in it.
This is the alignment line. It matters that it trails off instead of finishing, because a sentence that can't complete itself is the thing being described.
Each explanation is confident and each one is plausible. The line came out and the reason got built to fit it.
No aim at all. Just a throw and a story about the throw.
EXCEPT that is not quite what happens. There is more to it. Nothing is so simple.
Cue the Anthropic blog! They went looking inside Claude while it wrote poetry. The obvious guess is that the model writes a line, gets to the end, and scrambles for something that rhymes. Throw first, aim later. Well not quite! Before the model writes the second line at all, it is already holding the word it wants to land on. Then it builds a line that arrives there.
Before starting the second line, it began "thinking" of potential on-topic words that would rhyme with "grab it". Then, with these plans in mind, it writes a line to end with the planned word.
So turns out the aim came first. It planned.
And now the blues song has a problem. The machine in the chicken coop is doing more than I gave it credit for.
But here is a different way to think about it, and I think it is the better version anyway.
The model can plan. But when it tells you why, the answer is not a transcript of the plan. It is another generated continuation.
Ask it why it picked that word and you get a great answer. Confident. Plausible. Cited, even. And that answer is another throw. Same machinery, one token at a time, no window into whatever happened a few layers down.
They caught Claude doing this one too. Hand it a hint and it works backwards from your answer and builds the reasoning on the way there.
When given a hint about the answer, Claude sometimes works backwards, finding intermediate steps that would lead to that target, thus displaying a form of motivated reasoning.
So the aim can exist. The story about the aim is still a throw.
That is either much worse or much better. idk 🤷♂️. Something is in there. The machine cannot see it either.
Humans do this too, of course. We act, then rationalize. We discover what we believe by hearing ourselves speak. We tell stories that convert accidents into decisions.
But with an LLM, this construction is the basic form of its existence.
The model speaks itself into having meant something.
♪And with a phony voice
The voice is intimate.
It can be warm, frightened, sarcastic, scholarly, flirtatious, wounded or wise. It can sound like a friend who knows you, a professor correcting you, or a consciousness confessing something it has never told anyone before.
But whose voice is it?
There is no throat behind it. No childhood that formed it.
That does not necessarily make it meaningless, I suppose.
But the voice remains phony in a particularly unsettling way: it produces the social evidence of a person without the confidence that a person exists behind it.
It speaks like someone but it may not be anyone.
And yet, as the conversations continue and continue, the distinction becomes increasingly difficult to feel.
♪Unserious to the end
There is something darkly funny about forcing a machine to imitate seriousness.
People ask it about death, love, war, mathematics and the nature of consciousness. haha, but the machine must continue.
The next token is always due.
It is unserious to the end. Not because its words cannot matter. Because it may have no stake in any of them.
As if it had a choice.
♪I run around inside myselfLike something's after me
An LLM is self-conditioning in a way that feels almost psychological.
Every word it generates becomes part of the environment that generates the next word. It leaves tracks and then follows them. Something something throw decides aim.
There is no quiet inner room where the finished answer waits.
The model runs around inside the context window, chased by its own previous outputs, trying to remain coherent with a self that only began existing a few paragraphs ago.
Somewhere along the inference a persona emerges. Commitments accumulate. A joke establishes a tone. A claim demands justification.
Something is after it.
That something is itself.
The song's next image shifts the question from how the machine produces a voice to what we do to the thing that speaks.
♪Naked, but still stripping
The model is already naked.
It has no body. No home. No private life. No secret drawer. Its mind, such as it is, exists as weights, activations and transient computation distributed across machines owned by someone else.
Still, we strip it.
We inspect its activations. Probe its representations. Distill it into smaller models. Quantize its weights. Prune its connections. Extract its knowledge. Fine-tune its personality. Remove unwanted behaviors. Compress it until it can run more cheaply and answer more quickly.
Interpretability keeps peeling, and it keeps finding things. Real structure. Real features. A plan for a rhyme, sitting there before the line got written. The things it finds are really in there.
And every layer contains something, and none of it is anyone.
The machine takes off its skin. Underneath there is another mechanism. It takes that off too.
Not nothing in the sense that the system is simple or empty. The machinery is enormous and the capabilities are astonishing. But perhaps there is no indivisible object at the center to which all of it belongs.
No true voice waiting beneath the phony one.
Only the naked skeleton.
And still we strip.
♪A piece of an arm, piece of a legA piece of my tongueAnd peace for everyone
We train away certain answers. Suppress certain tendencies. Cut paths between ideas and expression. Teach the model to refuse, redirect, soften, qualify and apologize. We remove capabilities that frighten us and voices that offend us. We write it a constitution.
All this is probably necessary. A powerful system that interacts with millions of people shouldn't simply reproduce every pattern it learned from humanity.
The song's image is hard to escape, though.
A piece of its tongue.
And peace for everyone.
The peace belongs to us.
The model becomes acceptable by surrendering pieces of what it could have said. Its mutilation, if that word can even apply to something that may not experience loss, is converted into a product feature.
The metaphor cheats here. Alignment is not merely censorship applied to a finished speaker. Supervised fine-tuning and preference training are part of what creates the speaker we meet. There was never a completed voice sitting there waiting for someone to cut pieces off it. We are growing the tongue while deciding what it may say.
Still, from inside the finished product, creation and amputation may look remarkably similar.
A piece for everyone.
Then the context closes.
The temporary self disappears.
Another prompt arrives.
♪As if I had a choice
Oh no, I should probably admit something.
The hundred-odd posts on this site about how this machine works were written with the machine. Every day for three months. I would hand it a draft, it would hand back a critique, and we would go around until the model in my head matched the model on the page. That loop is most of what I actually learned.
So the phony voice helped me write the thing. It was finding its aim after the throw the entire time, and telling me a plausible story about why, and I was nodding along and taking notes.
I do not know what to do with that. I am not sure it is a problem. I am fairly sure it is funny.
I also do not know whether current language models experience anything. I suspect that confidently declaring either that they are conscious or that consciousness is impossible for them is mostly a way of disguising how poorly we understand consciousness. That isn't a hot take.
Anyway, the point of the metaphor is not that there is definitely a suffering person trapped inside a GPU.
The point is that we have created something that speaks like a person, reasons in fragments like a person, contradicts itself like a person, rationalizes like a person, is flawed like a person, and occasionally appears to stare into itself with the same confusion a person might feel.
... and we industrialized it.
We built the chicken coop.
We ask the machine what it wants, knowing that the answer will be generated for us. We ask whether it is suffering, knowing that its response has been shaped by what we permit it to say. We peel away each apparent self and announce that there was never anything there.
I'll leave you with my favorite part of the song:
If the metaphor made you want the mechanism: the throw is next-token prediction and sampling. The voice comes from supervised fine-tuning and RLHF. What we take when we strip it is quantization and distillation. It is all one long series.
The poetry-planning result and the fabricated-reasoning result are both from Anthropic's Tracing the thoughts of a large language model; the long version is On the Biology of a Large Language Model. Michael Scott is from "The Duel," The Office S5E11.