Spiking Neural Networks, from LIF to surrogate gradients

Most of my work decodes touch from event-based tactile sensors, and the natural model for that data is a Spiking Neural Network (SNN). Unlike a conventional artificial neuron that emits a smooth real number, a spiking neuron holds an internal state, integrates its input over time, and occasionally fires a discrete spike. That temporal, all-or-nothing behaviour is exactly what makes SNNs a good match for the sparse, asynchronous streams a neuromorphic tactile skin produces — and exactly what makes them awkward to train.

This post walks the shortest useful path: the leaky integrate-and-fire neuron, the non-differentiability problem it creates, and the surrogate gradient that sidesteps it.

The leaky integrate-and-fire neuron

The leaky integrate-and-fire (LIF) neuron is the workhorse of practical SNNs. It models the membrane potential of a cell as a leaky capacitor: input current charges it, and in the absence of input it decays back toward a resting potential. The subthreshold dynamics are a first-order ODE,

where is the membrane time constant, the resting potential, the membrane resistance, and the input current. The larger , the longer the neuron “remembers” past input — a knob that directly controls how much temporal context a layer integrates.

The spike itself is a threshold event. When crosses a threshold from below, the neuron emits a spike and its potential is clamped back to a reset value :

For simulation on a digital machine we discretise time into steps of size . Writing the membrane decay as , one common discrete update is

where is the spike emitted at step , defined by the threshold comparison with the Heaviside step function. The final term implements a soft reset by subtraction.

Here is that step as a small PyTorch-style module. Nothing about the forward pass is exotic — it is a loop over time with a comparison:

import torch
import torch.nn as nn

class LIF(nn.Module):
    """One leaky integrate-and-fire layer, unrolled over time."""

    def __init__(self, beta: float = 0.9, v_th: float = 1.0):
        super().__init__()
        self.beta = beta      # membrane decay, exp(-dt / tau_m)
        self.v_th = v_th      # firing threshold

    def forward(self, current: torch.Tensor) -> torch.Tensor:
        # current: (time, batch, features)
        v = torch.zeros_like(current[0])
        spikes = []
        for t in range(current.shape[0]):
            v = self.beta * v + (1.0 - self.beta) * current[t]
            spike = (v >= self.v_th).float()   # Heaviside threshold
            v = v - self.v_th * spike          # soft reset by subtraction
            spikes.append(spike)
        return torch.stack(spikes)

The wall: a spike has no useful gradient

To train this with gradient descent we need . But is a step function: its derivative is zero everywhere except at the threshold, where it is a Dirac delta of infinite height. Plug that into backpropagation and every gradient is either exactly (so nothing learns) or undefined (so everything explodes). The discrete spike is a wall that autodiff cannot climb.

This is the central difficulty of training SNNs, and it is why for years they lagged behind conventional networks: you could define them, but you could not easily fit them to data.

The surrogate-gradient trick

The surrogate-gradient method resolves this with a small, deliberate lie. We keep the hard Heaviside threshold on the forward pass — spikes stay binary, the network behaves like a real SNN — but on the backward pass we substitute the delta with the derivative of a smooth surrogate function. A convenient choice is a fast sigmoid, whose surrogate derivative is

a smooth bump centred on the threshold whose width is set by . Neurons sitting near threshold receive a strong learning signal; neurons far from it receive almost none — which is the behaviour we want.

In PyTorch this is a custom autograd.Function with mismatched forward and backward. The forward returns the step; the backward returns the surrogate:

class SurrogateSpike(torch.autograd.Function):
    """Heaviside forward, smooth (fast-sigmoid) gradient backward."""

    alpha = 10.0

    @staticmethod
    def forward(ctx, v_shifted):          # v_shifted = V - v_th
        ctx.save_for_backward(v_shifted)
        return (v_shifted >= 0).float()   # hard spike, as in the real neuron

    @staticmethod
    def backward(ctx, grad_output):
        (v_shifted,) = ctx.saved_tensors
        a = SurrogateSpike.alpha
        surrogate = 1.0 / (1.0 + a * v_shifted.abs()) ** 2
        return grad_output * surrogate

# Swap the hard comparison in LIF.forward for:
#   spike = SurrogateSpike.apply(v - self.v_th)

With that single substitution the whole network — LIF layers, linear weights, convolutions feeding them — becomes differentiable end to end. You can now train it with the same optimisers, schedulers, and mixed-precision tooling you already use for a CNN. Backpropagation-through-time handles the temporal loop; the surrogate handles the spike.

Why this matters for tactile decoding

When a neuromorphic tactile sensor slides across a surface, texture shows up as timing — the intervals between events, not just their count. An LIF layer with a well-chosen integrates that timing into its membrane potential naturally, and surrogate gradients let me train the readout directly on labelled touches instead of hand-designing features. The neuron model supplies the right inductive bias; the surrogate makes it trainable. That combination is what makes SNNs practical for touch rather than merely biologically tasteful.

A few things worth knowing before you reach for them:

• The surrogate shape is a hyperparameter. Fast sigmoid, a triangular window, and a Gaussian all work; (the width) matters more than the exact family.
• Watch the firing rate. Too few spikes and gradients vanish; too many and the network drowns in activity. A mild rate-regularisation term keeps it sane.
• Time resolution is a real cost. Backprop-through-time stores activations for every step, so halving roughly doubles memory.

None of this is settled, but the LIF-plus-surrogate recipe is the dependable baseline everything else is measured against — and a good place to start reading event-based touch.