Xsciens

WorldMAP NOW

WorldMAP NOW

Tag: Backpropagation

  • [Science] What Is Backpropagation? A Simple Guide to How AI Learns

    [Science] What Is Backpropagation? A Simple Guide to How AI Learns

    Backpropagation is the algorithm that lets artificial neural networks learn from their own mistakes, and it quietly powers much of today’s AI — from chatbots that answer your questions to the apps that recognize a face in a photo. Understanding it, even at a basic level, is one of the clearest ways to understand how modern deep-learning systems actually learn.

    Original Paper

    Title: Learning representations by back-propagating errors
    Authors: David E. Rumelhart, Geoffrey E. Hinton, Ronald J. Williams
    Journal: Nature, Vol. 323, pp. 533–536 (1986)
    DOI: 10.1038/323533a0

    What Is Backpropagation, Exactly?

    In plain terms, backpropagation — short for “backward propagation of errors,” also known as error backpropagation — is a method for training a neural network by repeatedly working out how wrong its answer was, and in which direction each connection should adjust. A neural network is built from layers of simple, interconnected units (loosely inspired by neurons) joined by adjustable connection strengths called weights. When a network is first created, its weights are usually set to small, carefully chosen random values, so its first guesses are generally poor. Backpropagation is the procedure that turns that rough starting point into a network that can recognize a cat in a photo, translate a sentence, or predict the next word in a reply.

    There is one important distinction worth keeping straight: backpropagation calculates how much each connection contributed to the error, while a separate step called gradient descent uses that information to actually adjust the connections. People often use “backpropagation” loosely to describe the whole training process, but technically the two are different steps working together.

    How the Algorithm Actually Works

    Training with backpropagation happens in two passes. In the forward pass, an input — a photo, a sentence, a row of sensor data — moves through the network’s layers in order, and the network produces a prediction. That prediction is then compared with the correct answer using a loss function, a numerical measure of how far the prediction was from the desired result.

    Backpropagation doesn’t tell a network what to think. It tells the network how wrong it was, and in which direction each connection should move to be a little less wrong next time.

    A simple analogy helps before the math: imagine a product moving through several stages of a factory line, and the finished item turns out defective. To find the cause, managers work backward from the end of the line, and each stage reports how much it may have contributed to the problem, passing that information to the stage before it. The backward pass in backpropagation does something similar, except it uses calculus to work this out exactly. Starting at the output layer and moving back toward the input, the algorithm applies the chain rule — a technique for tracking how a change at one stage ripples through a chain of dependent calculations. Each layer combines information about how its own calculations affected the next layer with the error information passed back from that layer, producing a gradient: a measure of how sensitive the loss is to a small change in each weight, at its current value.

    Diagram of the forward pass and backward pass in backpropagation
    The forward pass produces a prediction and a loss; the backward pass applies the chain rule, layer by layer, to compute gradients and update the weights.

    Once every weight has a gradient, gradient descent uses it to actually update the weights, nudging each one a small amount in the direction that reduces the error. Repeat this forward-pass/backward-pass/update cycle over thousands or millions of examples, and a network that started with randomly initialized connections gradually comes to reflect real structure in the data. The appeal of backpropagation is efficiency: it calculates gradients for every weight in a network in roughly the time needed for a small number of forward and backward passes, rather than requiring a separate experiment for each individual weight.

    Why Backpropagation Is Getting Attention Again

    Backpropagation is not new, and its mathematical core had already been explored before it became famous. Reverse-mode automatic differentiation — the calculus technique at the heart of the algorithm — traces back to Finnish researcher Seppo Linnainmaa’s 1970 master’s thesis, and American researcher Paul Werbos proposed applying a related idea to neural networks in his 1974 doctoral thesis. The method became particularly influential after David Rumelhart, Geoffrey Hinton and Ronald Williams published “Learning representations by back-propagating errors” in Nature in 1986, demonstrating that it could let multilayer networks automatically discover useful internal features rather than relying entirely on engineers to design them by hand.

    For years afterward, backpropagation-trained networks were mostly a research curiosity, limited by available data and computing power. That changed dramatically in 2012, when Alex Krizhevsky and Ilya Sutskever, working with their advisor Geoffrey Hinton, combined backpropagation, the large labeled ImageNet dataset, and graphics-card computing power in a network known as AlexNet. It dramatically outperformed competing approaches in that year’s ImageNet image-recognition competition, a result widely credited with triggering the modern deep learning boom. The large language models behind today’s generative AI chatbots are, at their core, still trained with variations of the same algorithm.

    Timeline of key milestones in the history of backpropagation, from 1970 to the 2020s
    Source: created based on Linnainmaa (1970), Werbos (1974), Rumelhart, Hinton & Williams, Nature (1986), and the AlexNet/ImageNet milestone (2012).

    What has put backpropagation back in the spotlight recently is a different question: does the brain do anything like this? In October 2026, a team at MIT’s McGovern Institute, led by Qianli Liao, Liu Ziyin and Yulu Gan with senior authors Mark Harnett and Tomaso Poggio, published a study in the Proceedings of the National Academy of Sciences proposing a computational model in which randomly connected artificial networks can self-organize their own forward and feedback pathways, using only local learning rules inspired by biological synapses and no global error signal — and still learn nearly as well as backpropagation on the tasks tested. The researchers are careful to frame this as a candidate mechanism and a testable hypothesis, not proof that real brains work this way, but it reopens a question that has nagged neuroscientists for roughly 40 years: backpropagation is clearly powerful enough to drive today’s AI, yet nothing in a biological brain has been shown to do precisely what it requires. (Related article: A Simulated Brain Circuit Builds Its Own AI-Like Learning System, MIT Finds.)

    Where You’ve Already Met Backpropagation

    Even though the math stays behind the scenes, the results of backpropagation-trained networks are part of ordinary life. Many modern face-unlock features and the photo apps that automatically group pictures of the same person run on neural networks trained with backpropagation or related gradient-based methods. Voice assistants that transcribe speech or answer spoken questions generally rely on networks trained the same way. Generative AI chatbots are also built using neural networks trained with backpropagation — during an early stage called pretraining, for example, a language model repeatedly tries to predict the next word in a passage of text, compares its guess with the actual word, and adjusts its weights to reduce the error; later training stages then refine how it follows instructions and responds to feedback.

    It’s worth keeping one distinction in mind: backpropagation mainly happens during training, not every time you use these tools. Once a network has learned, it typically uses its already-adjusted weights to make predictions, without recalculating gradients or updating itself each time — the heavy lifting happened earlier, often across billions of training examples.

    What to Watch Next

    Backpropagation itself is a mature, well-understood technique, so the open questions now sit in its limitations and the alternatives researchers are exploring. One is scientific: whether the MIT team’s self-organizing, locally-plausible learning circuits — or related approaches sometimes called feedback alignment or predictive coding — can be tested directly against real neural tissue, and whether they can match backpropagation’s performance on problems much larger than the ones tested so far. Another is practical: as AI models keep growing, researchers continue to look for training methods that need less computing power than standard backpropagation while keeping its accuracy. The next breakthrough may not come from making backpropagation itself faster, but from discovering how networks can learn without depending on it so completely — which is exactly the question AI researchers and neuroscientists are now asking together.

    Xsciens Editorial Note

    Backpropagation is best understood as a precise bookkeeping trick for calculating how a network should adjust itself, not as a theory of how any brain, artificial or biological, actually “thinks.”

    • A common misconception. “Neural network” learning is often described as though it copies how the brain learns, since artificial “neurons” were loosely inspired by biological ones. In practice, standard backpropagation’s backward calculation relies on precise weight-related information tied to the exact connections used for forward signaling — something no real neuron is known to reproduce or transmit in the same way. (Some alternative methods, such as feedback alignment, relax this requirement, which is itself a sign of how specific the original objection is.) The resemblance between artificial and biological “neurons” is a loose analogy, not a working description of what either one computes.
    • Limits and open questions. A common misreading is that backpropagation always needs a human-provided “correct answer,” like a teacher grading homework. That’s not quite right: it needs a differentiable training objective, which can come from supervised labels, but also from self-supervised tasks (such as predicting the next word in a sentence) or reinforcement-style rewards. What backpropagation does not explain, on its own, is why some network architectures generalize well to new data and others don’t, or how an AI system might decide what is worth learning in the first place — those remain active, only partly answered research questions, not settled ones.
    • What to watch next. The MIT team’s October 2026 study proposes, as a specific and testable hypothesis, that a four-synapse wiring motif linking ascending and descending pathways in the cortex could let real circuits self-organize a similar learning ability. This is a model prediction, not a confirmed feature of brain tissue — whether neuroscientists actually find that wiring pattern, and whether locally-plausible learning rules can match backpropagation’s performance on problems far larger than the ones tested so far, are the two developments worth watching.
    • From the editors — If brain-inspired, locally-learning AI matures the way this MIT team hopes, the assistants people use every day could stop feeling like blank tools that need re-explaining each time and start quietly adapting to how each person actually works. Getting there would likely mean systems that keep adjusting themselves after they leave the lab, learning from ordinary daily use instead of one big retraining cycle. That same ability to keep learning on its own is exactly what to watch: without a way to check what a system has quietly picked up, it could just as easily drift toward learning the wrong lesson from unusual or misleading experience. This is still an early computational demonstration, not a finished product — so if an assistant ever starts to feel like it “gets” you a little faster, it’s worth asking, every so often, what it actually learned to get there.

    References for this note: Quanta Magazine — Artificial Neural Nets Finally Yield Clues to How Brains Learn, arXiv — Brain-Inspired Machine Intelligence: A Survey of Neurobiologically-Plausible Credit Assignment

    Sources: IBM — What Is Backpropagation?, TechTarget — What Is a Backpropagation Algorithm?, Rumelhart, Hinton & Williams, Nature (1986), Seppo Linnainmaa, Computer History Museum — AlexNet Source Code, Xsciens — A Simulated Brain Circuit Builds Its Own AI-Like Learning System, MIT Finds

  • A Simulated Brain Circuit Builds Its Own AI-Like Learning System, MIT Finds

    A Simulated Brain Circuit Builds Its Own AI-Like Learning System, MIT Finds

    For decades, neuroscientists have faced a nagging puzzle. Modern AI owes its power to backpropagation, an algorithm that adjusts a network’s connections by propagating error information backward through precisely coordinated connections — a computation with no clearly established equivalent in biological circuitry. Yet some studies have found that activity patterns in trained artificial networks resemble those recorded in the brain’s own visual system, which only sharpens the question of how biological neurons, using only local signals and changing synapses, might approximate something similar. A new study from MIT’s McGovern Institute for Brain Research, published October 9 in the Proceedings of the National Academy of Sciences (PNAS), offers a candidate answer: when simple neuron-like units are wired randomly and left to follow local, biologically plausible plasticity rules, they can spontaneously self-organize into circuits that learn nearly as well as backpropagation does on the tasks tested — without anyone telling them how.

    原論文情報 / Original Paper
    Title: How Biological Synapses Self-Assemble Gradient Learning
    Authors: Qianli Liao, Liu Ziyin, Yulu Gan, et al.
    Journal: Proceedings of the National Academy of Sciences (PNAS)
    DOI: 10.1073/pnas.2606245123

    A 40-Year Mismatch Between Brains and AI

    Modern deep learning owes almost all of its success to backpropagation: a network makes a prediction, compares it with the correct answer, and then propagates that error backward, layer by layer, nudging every connection weight in the direction that would have reduced the mistake. It works brilliantly, but standard implementations depend on precise mathematical relationships between forward and backward connections, plus coordinated transmission of error information through the network. There is no established evidence that cortical circuits implement anything equivalent in its conventional form.

    The puzzle deepened as deep learning matured. Comparisons between the internal activity of backpropagation-trained networks and recordings from biological visual systems have repeatedly found intriguing similarities — not identical, but notably closer than chance would predict. Such similarities do not prove the brain runs backpropagation, but they raise an obvious question: if it does not, what local process could still produce computations that end up looking so much like it?

    Local Rules, No Blueprint

    The MIT team, led by Qianli Liao, Liu Ziyin and Yulu Gan with senior authors Mark Harnett and Tomaso Poggio (and co-author Brian Cheung), approached the question differently from most prior attempts to make backpropagation “more biological.” Instead of designing a circuit by hand and checking whether it could plausibly exist in a brain, they started from almost nothing: layers of neuron-like units connected at random, with no predetermined wiring diagram at all.

    The system runs on local synaptic plasticity rules, including heterosynaptic mechanisms that let a connection change in response to activity beyond that single synapse — a more specific relative of the familiar Hebbian idea that “cells that fire together wire together.” Critically, these rules only need information available locally, at or near a given synapse (the junction where one neuron signals to the next). No unit needs to know what is happening elsewhere in the network, and no single global error signal is broadcast to the whole system.

    Diagram showing random wiring self-organizing into a learning circuit
    Source: created based on findings from MIT McGovern Institute / PNAS (2026).

    Circuits That Wire Themselves

    Left to run under those local rules, the randomly connected units did not stay random for long. Over repeated rounds of activity, the network developed small, recurring wiring patterns the researchers call “self-assembling motifs.” The model’s analysis points to a specific four-synapse motif linking the brain’s ascending, bottom-up pathways with its descending, top-down pathways — a concrete structural prediction that experimentalists can now go looking for in real neural tissue, not a pattern already confirmed to exist there.

    What these motifs do next is the more remarkable part: networks of them self-organize further, so that their combined dynamics approximate a generalized form of stochastic gradient descent — the family of optimization methods that includes backpropagation — without a hand-designed, globally coordinated algorithm directing them. In other words, before the network learned anything from its training data, it first had to build the machinery that would let it learn.

    Starting from random connectivity, local synaptic plasticity rules were enough for the network to assemble circuit motifs whose combined dynamics approximate gradient-descent-style learning.

    Holding Its Own Against Backpropagation

    A biologically plausible circuit is only interesting if it actually works, so the researchers tested their self-organized networks on image pattern recognition and classification tasks. According to MIT, the self-assembled circuits achieved performance comparable to conventional networks trained directly with backpropagation on the tasks tested. The MIT News announcement does not report specific accuracy figures or benchmark datasets — those details sit in the full PNAS paper — so the result is best read as evidence that self-organization carried no obvious cost on this particular test, not as a claim that it matches backpropagation across deep learning in general.

    That matters because it suggests biological plausibility and computational competence are not necessarily in tension. For years, biologically inspired alternatives to backpropagation have often underperformed it on real tasks. A circuit that assembles itself from random wiring and still competes with backpropagation on the tasks it was tried on narrows that gap, at least in this one demonstration.

    A Prediction Neuroscientists Can Go Test

    The researchers are careful to note that a self-organizing network learning like this does not prove the human brain learns this way — it is a candidate mechanism, not a confirmed one. But unlike many theoretical proposals in computational neuroscience, this one comes with a concrete, falsifiable prediction: if the brain really does something like this, cortical circuits should contain that specific four-synapse motif connecting ascending and descending pathways. Experimental neuroscientists can now search actual brain tissue for exactly that wiring signature, which is what turns an elegant simulation into a testable scientific hypothesis.

    Why It Matters Beyond the Lab

    If the prediction holds up, the payoff would extend in both directions. For neuroscience, a confirmed biologically plausible learning circuit would help explain how the brain adapts throughout life, and could sharpen our understanding of what goes wrong in conditions involving altered learning or memory, as well as inform the design of future brain-computer interfaces that need to interact with the brain’s own learning rules rather than fight against them.

    For AI, the lesson cuts the other way: today’s networks are built top-down, with every connection and training rule specified by engineers before a single example is seen. A system that can bootstrap its own learning machinery from random wiring and local plasticity rules hints at an alternative path — architectures that configure themselves as they encounter data, rather than ones that are handed a finished blueprint. Liao, Ziyin, Gan and their colleagues frame their result as an early step in that direction, not a finished product, but their work gives both neuroscientists and AI researchers a concrete new question to test: not just how networks learn once they are built, but how learning itself can come preinstalled.

    For more on where artificial and biological intelligence still diverge, see our earlier coverage of how MIT neuroscientists found the brain re-files information the instant you make a decision, a separate finding that is likewise reshaping how AI researchers think about adaptive, flexible learning.

    Sources: MIT News, Proceedings of the National Academy of Sciences (PNAS), arXiv preprint