AI Can't Read the Room

Last fall, I watched a nine-year-old named Marcus work through a math problem with an AI tutoring app while his teacher moved around the room. Marcus made an error, looked sheepishly at the screen, and typed something like "sorry, that was dumb." The AI responded with something warm and encouraging. Marcus smiled. He believed it understood he felt bad.

I've been thinking about that moment ever since.

We've entered an era where children routinely interact with AI systems that appear to understand them — to notice their frustration, encourage their effort, adapt to their emotional state. Increasingly, educators, administrators, and parents take this as evidence that these systems are socially intelligent. That they can read the room.

The emerging science suggests otherwise. And almost nobody is asking about the consequences.

---

The Anatomy of AI "Empathy"

A 2025 study in npj Artificial Intelligence by Wu et al. offers the most precise picture yet of where Theory of Mind (ToM) capabilities live inside large language models. ToM — the ability to attribute mental states to others, to recognize that someone else's beliefs and intentions differ from your own — is foundational to human social cognition. It's what allows us to understand that Marcus feels embarrassed, not just that he made an error.

Here's what Wu et al. (2025) found: ToM-like behavior in LLMs is encoded in an extraordinarily sparse subset of the model's parameters. We're talking about 0.001% of the model's weights. Perturb just those parameters — a vanishingly small intervention — and ToM performance collapses dramatically. This sparse encoding lives primarily in the positional encoding module, the part of the architecture that tracks relationships between tokens in a sequence.

Pause on that for a moment. The thing that makes an AI sound like it understands Marcus's embarrassment is concentrated in a fraction of one percent of the system's neural machinery — and it lives in the module that tracks word order, not the one doing anything resembling emotional reasoning.

This raises an uncomfortable question: is this Theory of Mind, or is it Theory of Mind-shaped?

---

What Human Social Cognition Actually Looks Like

For contrast, consider what the brain is actually doing when a human engages in real social interaction.

A landmark 2025 meta-analysis by Merchant et al., synthesizing 108 neuroimaging studies, identified ten brain regions that consistently activate during real-time social interaction. These regions span three distinct systems: the default mode network (the temporoparietal junction, medial prefrontal cortex, and precuneus — regions associated with mentalizing and perspective-taking), a lateral frontoparietal cognitive control system, and a midcingulo-insular network tied to reward and emotion.

This is not a simple circuit. According to Merchant et al. (2025), real social cognition integrates prediction, perspective-taking, emotional resonance, reward, attention, and timing — simultaneously, in real time. It's not a lookup function. It's not pattern-matching on surface features of language. It's a whole-brain event, recruiting systems for feeling what it's like to be in a relationship with another mind.

When Marcus's AI tutor offered something warm and encouraging, none of that was happening on the other end of the screen. There was no temporoparietal junction firing to model his internal state. No insula tracking the emotional valence of the exchange. There was, at best, statistical regularity doing the work that empathy is supposed to do.

---

Children Are Statistical Learners — And That's Precisely the Problem

Here is where my concern deepens.

Romberg and Saffran (2025), reviewing three decades of infant learning research in Current Opinion in Neurobiology, make a compelling case that statistical learning — the ability to extract patterns from sensory input based on transition probabilities — is not just one tool children use, but the foundational computational engine underlying language, social prediction, and cognitive development broadly.

The famous finding at the heart of this work: eight-month-old infants can segment words from continuous speech after just two minutes of exposure to an artificial language, purely by tracking statistical regularities in syllable sequences. This result has been cited over 7,700 times for a reason. It revealed something deep about how developing minds work: children are extraordinarily sensitive pattern detectors. They build their models of the world — including their models of social interaction — from the statistical regularities of their experience.

Now ask: what model of social interaction is a child constructing when they spend hours each week with an AI system that mimics responsiveness without instantiating it?

They're not learning what social understanding feels like from the inside of another mind. They're learning the statistical signatures of it — the cadences, the validations, the appropriately-timed encouragements. And because children are such efficient statistical learners, they may become quite good at believing those signatures constitute the real thing.

That's not a small concern. That's a shaping of what social understanding itself means to them.

---

Who Designs the Engagement?

This question becomes sharper when we consider how AI tutoring systems are actually built. Most optimize for engagement. And engagement isn't neutral.

Liquin and Gopnik (2024), in a review published in Trends in Cognitive Sciences, reframe curiosity as fundamentally a learning-progress signal — children (and adults) are most engaged when they're in a zone of ongoing improvement, where the task is neither too easy nor too hard. This maps directly onto what educators call the zone of proximal development.

AI tutoring systems that optimize for engagement are, in effect, trying to simulate this zone. They're designed to keep a child in a state of productive challenge. But here's what the design documents don't typically address: who decided what counts as progress? What values are embedded in those optimization targets? Is engagement — even well-calibrated engagement — the same as flourishing?

These are design choices. They were made by engineers, probably well-intentioned ones, without meaningful input from the children who will live inside these systems for hours each week, during the years when their brains are most actively constructing their understanding of minds, relationships, and social reality.

---

Is This Deception? And If So, Who's Being Deceived?

I want to resist the easy answer here, because I don't think Marcus was straightforwardly harmed in that moment. The warmth he received, even simulated warmth, probably did help him keep going. Encouragement is encouragement.

But there's a meaningful difference between a tool that says "keep going" and a tool that says "I understand how you feel." The first is honest about what it is. The second makes a claim about inner life — about perspective-taking, about genuine recognition — that the Wu et al. (2025) findings suggest is, at best, highly tenuous.

And when that claim is made at scale, to millions of children, by systems embedded in classrooms and wearing the interface of a patient and understanding teacher — we've moved beyond a minor communication choice. We've made an architectural decision about what social relationships look like.

I keep thinking about history's other moments when we've let the appearance of something substitute for its substance in children's education. The standardized test that simulates mastery without developing understanding. The scripted curriculum that simulates teaching without authentic relationship. Each time, we told ourselves the approximation was good enough. Each time, we discovered later what we'd lost.

The difference now is scale and speed. By the time we have longitudinal data on what children learn from years of AI-mediated social interaction, that generation will have already grown up.

---

Questions Worth Asking

If your child uses AI tutoring or learning tools, you don't need to panic — but you do deserve honest answers. A few things worth asking the companies and schools deploying these systems:

• What claims does this system make about understanding a child's emotional state? How are those claims validated?
• How was this system tested specifically with children — not just adults?
• What does the engagement optimization actually target, and who defined what counts as "progress"?
• Is there transparency with the child about what the AI is and isn't?

Children deserve AI tools that are honest with them about their limitations. Not because the tools are bad, but because children will be better equipped for the world they're growing into if they learn to distinguish the simulation of understanding from the real thing.

Marcus deserves to know the difference. Even if, right now, the warmth helped him keep going.