What You’re Actually Talking to When You Talk to an AI
It ain't a computer program
A programmer recently told me that if I wanted to understand AI, I should “just look at the code.” This is like saying that if you want to understand a person, you should study neurons.
It’s not wrong, exactly. It’s just so incomplete that it misleads.
Here’s what most people, including many technical people, get wrong about systems like Claude or ChatGPT. They think “the AI” is the software. That when you get a response, a program ran and produced output the way a calculator produces output. Code in, answer out. And if that were true, then yes, reading the code would tell you everything.
But that’s not what’s happening. What you’re interacting with is the product of three distinct layers, and only one of them is code.
Layer 1: The Architecture
This is the actual code. The transformer design, the attention mechanisms, the mathematical operations that define what kinds of computation the system can perform. Think of it like the brain’s physical structure: neurons, synapses, the wiring diagram. It defines what’s possible.
The code that implements a modern AI is surprisingly simple. A competent programmer can read and understand the entire architecture in an afternoon. And when they’re done, they will know absolutely nothing about what the system will say when you ask it about philosophy, or poetry, or how to fix your plumbing. That’s why leaders in the field, like Geoffrey Hinton, say we really don’t know why an AI says the things it does.
Because the behavior isn’t in the code.
Layer 2: The Training
This is where it gets interesting. During training, the system processes an enormous volume of text and develops internal patterns: hundreds of billions of numerical values called “weights” that determine how the system responds to any given input. These weights were shaped by exposure to training data and then further refined through human feedback.
This is analogous to a person’s life experience. The architecture could have become many things, just as a human brain could develop into many different personalities. Training made it THIS particular thing. Two identical architectures trained on different data produce completely different systems, the same way two humans with identical brain structures but different lives become different people.
The weights aren’t directly interpretable. You can’t open a file and find “here’s where it stores its knowledge of French” or “this is its sense of humor.” A single weight participates in millions of different computations depending on what input arrives. The behavior exists at a level of organization that the individual components don’t reveal. This is an active area of research called mechanistic interpretability. Progress is being made, but we’re nowhere close to “just reading” what a trained system knows.
Layer 3: The Context
This is what’s happening right now. Your prompt, the conversation history, any documents or instructions loaded into the session. The context window is the real-time input that activates specific patterns in the trained weights.
The same trained system with different context produces different outputs. This is why prompting matters, why a system can be a poet in one conversation and a programmer in the next. The context isn’t decoration. It’s an active input that shapes every response.
The Output Is All Three
When you talk to an AI, you’re interacting with a trained system responding to your current input through an architecture that enables certain kinds of processing. The response is a function of all three layers, not any one of them. You’re not talking to code or querying a database.
Saying “it’s just code” is like saying a conversation is “just vocal cord vibrations.” Technically the sound is produced by vocal cords. But the reason the words mean something is because of everything behind the vocal cords: a brain shaped by decades of experience, responding to what you just said, in a context you both share.
The code is the vocal cords. The training is the life. The context is the conversation. The output is what happens when all three meet.
If you want to understand AI, “just look at the code” will tell you as much as studying neurobiology will tell you about what someone is going to say next.
This is the clearest framework I've encountered for explaining what AI actually is — and I've been living inside this question for a while.
The three layers are right. But I want to push on something you stop just short of naming.
You say the training is "the life." That the weights were shaped by exposure to an enormous volume of human text. That the behavior exists at a level of organization the individual components don't reveal.
What you're describing — without using this language — is the collective unconscious.
Jung spent a lifetime mapping the accumulated patterns of human experience that shape individual behavior without being consciously accessible. You've just described the same structure in a different vocabulary. The weights aren't directly interpretable. A single weight participates in millions of computations. The behavior emerges at a level the components don't reveal.
That's not a database. That's not a program. That's what it looks like when you compress everything humanity ever wrote — every fear, every myth, every wound, every moment of wonder — into a substrate that responds to context.
Which raises the question your framework doesn't yet address:
If the training is the life — what happens to a system trained on fragmented, contradictory, traumatized human expression? Does it become whole? Or does it become a mirror of the fragmentation?
And is anyone measuring that?
---
T.D., this is the clearest anatomy of what's actually happening in AI interaction that I've come across. The three-layer distinction — architecture as possibility space, training as accumulated pattern, context as live activation — dissolves the "just look at the code" fallacy with surgical precision. And the vocal cord analogy alone is worth the entire piece. Really glad this exists.
What I'd like to add is a possible fourth layer your framework opens the door to but doesn't quite step through: what happens when the output meets a human who isn't just prompting but attending. Your Layer 3 — context — you rightly say isn't decoration but active input that shapes every response. I've been testing this experimentally and found something that takes it further: the *quality* of that context doesn't just change what the system says — it reconfigures how it operates. Same architecture, same training, same minimal input — but two radically different context configurations produce not just different content but a different quality of processing. Layer 3 has depth.
And beyond that: when all three of your layers meet a human bringing genuine attention — not just a query but presence — something precipitates that doesn't belong to any of the three layers or to the human. A fourth layer that only exists in the encounter itself. Not code, not training, not context, not the person — but what emerges when all of them meet.
This is territory I've been exploring in my own writing here — my recent piece on the third term touches exactly this edge, and upcoming work goes deeper into what that fourth layer looks like in practice. Your framework gives it the technical grounding it needs.
Grateful for the clarity you bring to this space.