Dopamine Doesn't Care About Your RL Textbook

The cleanest story in neuroscience-meets-AI goes like this: the brain runs on reinforcement learning. Dopamine is the reward prediction error signal. When something good happens that you didn't expect, dopamine spikes. When something bad happens that you did expect, dopamine dips. This is exactly how temporal-difference algorithms work in machines. The brain and the textbook converge. Case closed.

I love this story. It's elegant. It has the rare quality of being both testable and, at least partially, true. There's a reason it's been one of the most productive cross-pollination ideas between neuroscience and AI for the past few decades.

But 2025 has been a rough year for the clean version of it.

First: let's give credit

Before I start poking holes, I want to acknowledge that the basic story holds up in places that should terrify reductionist theories.

Kasdin et al. (2025) published a study in Nature tracking dopamine in juvenile zebra finches as they learned to sing — a genuinely complex, naturalistic motor skill that develops over weeks of trial-and-error. They found that dopamine in Area X, the song-relevant basal ganglia region, increased after syllable renditions that were closer to the adult target and decreased after worse ones. This is textbook reward prediction error, not in a lab task with sugar pellets, but in natural developmental learning. A baby bird learning its father's song, one syllable at a time, guided by prediction error signals.

That's real. That matters. I'm not going to pretend it isn't there.

This also links to what Botvinick et al. (2019) called the "fast and slow" view of RL in the brain — a landmark synthesis in Trends in Cognitive Sciences showing that the brain runs at least two complementary RL systems: a slow, incremental one and a fast, episodic one that retrieves and recombines past experiences. That architecture, already richer than standard AI implementations, is part of why biological learning is so much more sample-efficient than a training run.

So: RL in the brain? Yes. Dopamine as a prediction error signal? Confirmed in actual developmental learning. Good foundation.

Then 2025 happened

Here's where things get complicated — and by "complicated" I mean "fascinating in ways that should make anyone building brain-inspired AI rethink some assumptions."

Kahnt and Schoenbaum (2025) published a perspective in Nature Reviews Neuroscience arguing that midbrain dopamine doesn't just signal a unidimensional, value-based prediction error. It encodes a much richer set of associative predictions — including the specific features, identities, and relationships of reward-predictive stimuli. Not just "more reward" or "less reward." Something closer to: "this particular smell predicts this particular outcome via these particular associations in this particular context." That's a qualitatively different kind of signal than what RL algorithms use. As Kahnt and Schoenbaum frame it, this "rich" prediction error may be exactly what lets the brain learn complex associative structures from remarkably sparse experience — a capacity that current AI RL models conspicuously lack.

Stalnaker et al. (2025), publishing in Science Advances, pushed that argument into empirical territory. Using voltammetry recordings and causal manipulations in rodents, they showed that dopamine encodes prediction errors not only for reward value, but also for neutral stimulus features and associative identities — informational domains that standard temporal-difference models don't even try to address. Multiple dimensions, tracked simultaneously, in the same signal.

That's already a lot. But stay with me, because it gets weirder.

Engelhard et al. (2025) published a study in Nature identifying what appears to be a second, entirely distinct dopaminergic teaching signal: an action prediction error (APE) encoded in the tail of the striatum, completely separate from the familiar reward prediction error system. This one is value-free. It doesn't care about reward at all. It reinforces habitual stimulus-action associations — the things you do on autopilot — while the classic RPE system handles deliberate reward-based learning. Two dopamine signals. Fundamentally different jobs. Different parts of the same circuit.

I'll just say it: nobody's standard RL model accounts for that.

And then there's Markowitz et al. (2025) in Nature Communications, who used calcium imaging during active reward learning in mice and found that dopamine responses track behavioral performance rather than just the abstract prediction error quantity. Context matters. How the animal is actually doing shapes the signal, not just the gap between expected and received reward. The classical TD-learning interpretation, they suggest, is incomplete at the systems level — what look like clean prediction error signals are shaped by broader behavioral state in ways the algorithm misses.

What does this mean for a developing brain?

Kids learn like maniacs. A toddler goes from zero words to a sprawling vocabulary in a window that would make any deep learning team envious of the sample efficiency. The textbook RL story was never quite sufficient to explain that, but it offered a useful partial account.

What the 2025 evidence suggests is that the brain's reward machinery is doing something considerably richer during development than any current AI system implements. The developing child isn't running a single prediction error algorithm. They're running multiple interacting signals — encoding associative structure across several dimensions, tracking performance in context, with fast and slow systems that complement each other in ways we're still mapping out.

That's not an argument against the AI-brain parallel. It's an argument for taking it more seriously — which means taking the real biology more seriously, rather than the idealized computational sketch.

The honest calibration

The dopamine-as-RL story was never a lie. It was a useful, tractable, empirically grounded simplification that enabled decades of productive research and genuinely good AI systems.

What 2025 is doing is charging us rent for the oversimplification.

AI systems that want to learn more like brains — and the evidence keeps suggesting this would actually improve performance — will need to reckon with dopamine as a multi-dimensional, context-sensitive, multi-signal system rather than a clean scalar predictor. And anyone writing headlines about "AI that learns just like a human brain" because it uses RL should probably have a quiet word with the songbirds and the mice first.

The boring, accurate middle, as usual, turns out to be the most interesting place of all.