Kiru Lab / Deep Learning / From Neuron to Network
The Artificial Neuron and Its Honest Limits
A weighted sum and a threshold. The metaphor to biology is real, thin, and worth stating precisely.
Concept · about 25 minutes
The artificial neuron computes a weighted sum of its inputs, adds a bias, and passes the result through a nonlinear function. McCulloch and Pitts proposed it in 1943 as an abstraction of a biological neuron, and it has barely changed since.
What the metaphor gets right
- Both integrate many weighted inputs into one output.
- Both have a threshold-like response rather than a purely proportional one.
- In both, the connection strengths — not the units — carry what was learned.
What it gets wrong
- Biological neurons communicate in spike trains over time; artificial ones emit a single static number.
- Dendrites perform local nonlinear computation before anything reaches the soma; the artificial model sums linearly.
- Neuromodulators like dopamine and serotonin globally reconfigure how a whole circuit behaves; there is no equivalent knob in a standard network.
- Real synapses update from locally available signals; backpropagation requires a global error signal routed backward along the exact forward weights, which biology has no clear mechanism for.
The Bio Mirror entry on the neuron and the synapse holds each of these differences up deliberately. The divergences are more informative than the similarities: they are a list of computational strategies biology found and we have not adopted.
Use the metaphor as a source of hypotheses, never as evidence. "The brain does it this way" is a reason to try something, not a reason to believe it works.
Hold on to
- The artificial neuron is a weighted sum plus a nonlinearity
- Learning lives in the connections, in both systems
- The differences from biology are a research agenda, not a footnote
Work through
Try each one before opening the solution. Getting it wrong first is most of where the learning happens.
-
Implement a single neuron and train it to compute AND, then OR, then fail at XOR. Explain the failure geometrically.
Hint
AND and OR are linearly separable in the plane. Plot XOR's four points and try to draw one line.
Solution
A single neuron computes a weighted sum against a threshold, which geometrically is one straight line dividing the plane. AND and OR each have a line that separates their classes. XOR's positive cases sit on opposite corners, so no single line works — you would need two. This is Minsky and Papert's 1969 result, and it stalled the field for a decade until multilayer training became practical.
import numpy as np X = np.array([[0, 0], [0, 1], [1, 0], [1, 1]], dtype=float) def train_perceptron(X, y, epochs=100, lr=0.1): w, b = np.zeros(X.shape[1]), 0.0 for _ in range(epochs): for xi, target in zip(X, y): pred = 1.0 if xi @ w + b > 0 else 0.0 w += lr * (target - pred) * xi b += lr * (target - pred) return w, b for name, y in (("AND", np.array([0, 0, 0, 1.0])), ("OR", np.array([0, 1, 1, 1.0])), ("XOR", np.array([0, 1, 1, 0.0]))): w, b = train_perceptron(X, y) acc = np.mean(((X @ w + b) > 0).astype(float) == y) print(f"{name}: accuracy {acc}") # XOR stalls at 0.5 or 0.75 -
Write down three predictions the neuron metaphor makes that turn out to be false in real networks.
Hint
Think about timing, dendrites, modulation, and how learning signals reach a synapse.
Solution
Three false predictions the metaphor invites. (1) That a unit's output is a meaningful scalar "firing rate" — in practice a unit is one arbitrary coordinate of a superposed representation, so reading it alone is misleading. (2) That individual units specialize for interpretable concepts — polysemanticity says otherwise. (3) That the network learns the way a brain does — backpropagation requires a global error routed backward along the exact forward weights, which has no established biological mechanism. Each prediction fails, and each failure is more instructive than the resemblance that suggested it.
Sign in to track your progress through the lab.