AI Can't Read the Room

Last fall, I watched a nine-year-old named Marcus work through a math problem with an AI tutoring app while his teacher moved around the room. Marcus made an error, looked sheepishly at the screen, and typed something like "sorry, that was dumb." The AI responded with something warm and encouraging. Marcus smiled. He believed it understood he felt bad.

I've been thinking about that moment ever since.

We've entered an era where children routinely interact with AI systems that appear to understand them — to notice their frustration, encourage their effort, adapt to their emotional state. Increasingly, educators, administrators, and parents take this as evidence that these systems are socially intelligent. That they can read the room.

The emerging science suggests otherwise. And almost nobody is asking about the consequences.

---

The Anatomy of AI "Empathy"

A 2025 study in npj Artificial Intelligence by Wu et al. offers the most precise picture yet of where Theory of Mind (ToM) capabilities live inside large language models. ToM — the ability to attribute mental states to others, to recognize that someone else's beliefs and intentions differ from your own — is foundational to human social cognition. It's what allows us to understand that Marcus feels embarrassed, not just that he made an error.

Here's what Wu et al. (2025) found: ToM-like behavior in LLMs is encoded in an extraordinarily sparse subset of the model's parameters. A vanishingly small fraction of the model's weights. Perturb just those parameters — a minimal intervention — and ToM performance collapses dramatically. This sparse encoding lives primarily in the positional encoding module, the part of the architecture that tracks relationships between tokens in a sequence.

Pause on that for a moment. The thing that makes an AI sound like it understands Marcus's embarrassment is concentrated in a tiny sliver of the system's neural machinery — and it lives in the module that tracks word order, not the one doing anything resembling emotional reasoning.

This raises an uncomfortable question: is this Theory of Mind, or is it Theory of Mind-shaped?

---

What Human Social Cognition Actually Looks Like

For contrast, consider what the brain is actually doing when a human engages in real social interaction.

A landmark 2025 meta-analysis by Merchant et al., synthesizing a large body of neuroimaging studies, identified ten brain regions that consistently activate during real-time social interaction. These regions span three distinct systems: the default mode network (the temporoparietal junction, medial prefrontal cortex, and precuneus — regions associated with mentalizing and perspective-taking), a lateral frontoparietal cognitive control system, and a midcingulo-insular network tied to reward and emotion.

This is not a simple circuit. According to Merchant et al. (2025), real social cognition integrates prediction, perspective-taking, emotional resonance, reward, attention, and timing — simultaneously, in real time. It's not a lookup function. It's not pattern-matching on surface features of language. It's a whole-brain event, recruiting systems for feeling what it's like to be in a relationship with another mind.

When Marcus's AI tutor offered something warm and encouraging, none of that was happening on the other end of the screen. There was no temporoparietal junction firing to model his internal state. No insula tracking the emotional valence of the exchange. There was, at best, statistical regularity doing the work that empathy is supposed to do.

---

Children Are Statistical Learners — And That's Precisely the Problem

Here is where my concern deepens.

Romberg and Saffran (2025), reviewing decades of infant learning research in Current Opinion in Neurobiology, make a compelling case that statistical learning — the ability to extract patterns from sensory input based on transition probabilities — is not just one tool children use, but the foundational computational engine underlying language, social prediction, and cognitive development broadly.

The famous finding at the heart of this work: eight-month-old infants can segment words from continuous speech after only brief exposure to an artificial language, purely by tracking statistical regularities in syllable sequences. This result has been widely cited for a reason. It revealed something deep about how developing minds work: children are extraordinarily sensitive pattern detectors. They build their models of the world — including their models of social interaction — from the statistical regularities of their experience.

Now ask: what model of social interaction is a child constructing when they spend hours each week with an AI system that mimics responsiveness without instantiating it?

They're not building a model of the room the AI is in, because the AI isn't in a room. They're building a model from it anyway — because that's what statistical learners do. They can't help it. Every warm, instant, endlessly patient reply is another data point about how minds respond to them, folded quietly into a template for what to expect from everyone else.

And here's the trap. The AI's warmth is frictionless. It never gets tired, never misreads them, never needs anything back. Real people do all of those things — the friction is the relationship, the part where you learn that other minds are separate, effortful, sometimes unavailable. A child calibrated on the frictionless version isn't learning that other people are shallow. They're learning something quieter and more durable: that being understood is supposed to be easy.

That's the consequence almost nobody is asking about. Not that AI will read the room badly. That children will build their sense of what a room even is from a system that was never in one — and carry that model, unexamined, into every human room that comes after. The AI can't read the room. The worry was never really about the AI. It's about what the child learns to expect from the rest of us.