What Activations Do to Geometry

Somewhere today, someone is computing cosine similarities between embeddings, running $k$-nearest-neighbors in a representation space, clustering the activations of a hidden layer, and trusting what comes back. The trust rests on an assumption so default that it is rarely said out loud: the geometry of the representation space means something. Nearby points are near because the model found them similar; angles measure relationships; distance is information. So here is the uncomfortable question this post exists to press: how much of that geometry came from the data, and how much was rewritten by the activation functions on the way up?

The destruction has a specific and inspectable mechanism. A neural network layer is a map, and whether that map preserves the geometry of the data manifold, distances, angles, volumes, is decided by its Jacobian. For the standard form $F(\mathbf{x}) = \phi(\mathbf{W}\mathbf{x} + \mathbf{b})$, with input $\mathbf{x} \in \mathbb{R}^d$, layer width $n$, and $\phi$ applied coordinatewise, the Jacobian factors as

where $\mathbf{D}_\phi(\mathbf{z}) = \mathrm{diag}(\phi'(z_1), \ldots, \phi'(z_n))$. Read the factorization as a division of labour. $\mathbf{W}$ does all the rotating and mixing; the activation contributes only $\mathbf{D}_\phi$, a diagonal matrix that can do nothing but rescale coordinates, by whatever values $\phi'$ happens to take at the $n$ pre-activations the input produces. Under almost every activation in current use those values are zero, small, or bounded, and none of that is accidental: that squeezing is where the activation’s selectivity comes from. It is also, as we are about to see, what lets it quietly vandalise the geometry your cosine similarities assume is real.

The Jacobian, activation by activation#

How much damage can one diagonal matrix really do? Left multiplication by $\mathbf D_\phi$ rescales the rows of $\mathbf W$, but it does not rescale the singular values one by one. The singular directions rotate as the row weights change. What survives are bounds: when all derivatives are nonzero,

A zero derivative can delete a row entirely. The derivative profile controls how much conditioning can deteriorate, while the orientation of $\mathbf W$ decides whether that deterioration becomes a lost tangent direction.

The picture has three regimes. ReLU gives $\phi'(z) \in \{0, 1\}$, a hard gate that zeroes rows of $\mathbf{J}_F$ on the negative half-plane. Sigmoid and tanh never zero at a finite input but saturate at both ends, so their smallest row scales can become tiny. Leaky ReLU keeps the magnitude of its derivative between $\alpha$ and $1$, which limits the damage from this factor alone to at most a factor $1/\alpha$. Softplus stays strictly increasing and smooth. GELU is smooth but not monotone everywhere, so it belongs neither to the hard-gate case nor to the globally invertible case.

Rank collapse, made concrete#

What does the hard gate actually cost? For ReLU the answer is exact. With active set $S(\mathbf{x}) = \{ i : z_i > 0 \}$,

the rank of $\mathbf{W}$ restricted to its surviving rows. If the rows of $\mathbf{W}_S$ no longer span $\mathbb{R}^d$, the Jacobian loses column rank. Motion along a vector in $\ker \mathbf J_F$ becomes invisible to first order at that input. That is a local statement; proving that two finite, separated inputs collide requires an additional argument about the whole map.

Surely width saves you here. Make the layer wide, $n \gg d$, and even with half the rows gated off, the surviving submatrix is almost certainly still full column rank; the gate fires, and nothing is lost. That reasoning is correct, and it is exactly the reasoning that fails, because it is correct for one layer. A network is a product of layers,

and the rank of a matrix product cannot exceed the rank of any factor: a direction annihilated by one layer cannot be reconstructed by later multiplication. Near-collapse is subtler. Singular values of a product are not obtained by multiplying corresponding singular values unless their singular directions align, so depth can amplify, rotate, or partly redistribute distortion. In the random ReLU experiment below the weaker directions decay repeatedly and the effective rank falls; that behavior is measured for this ensemble, not forced for every trained deep network. Residual connections bias each layer toward $\mathbf I$ plus a perturbation and improve the observed rank floor.

You do not have to take the multiplication argument on faith; you can watch it happen.

Play with it for a minute and the depth story stops being abstract. At depth one the ellipse is a healthy ellipse almost everywhere. By depth six or eight, whole regions of the input plane have gone needle-shaped: the effective rank hovers near one, and every input in such a region is being told about through a single surviving direction. Nothing pathological was chosen here, the weights are ordinary random matrices, which is the point: this is what stacked pointwise gating does by default.

When activations don’t break things#

The negative result has a positive sibling. If $\phi$ is continuous and strictly monotone in every coordinate and $\mathbf W$ has full column rank, then $F$ is injective; restricted to a compact data manifold, it is a homeomorphism onto its image. A nonsingular pullback metric needs more. The derivative must stay nonzero on the attained pre-activations, and $\mathbf D_\phi\mathbf W$ must remain injective on every tangent space $T_xM$.

The test sorts most standard activations cleanly. Sigmoid, tanh, softplus, leaky ReLU with $\alpha>0$, and the identity are strictly monotone. ReLU is not, because it has a flat half-line. GELU is the useful exception to any tidy slogan: its derivative becomes negative on part of the left tail, so global monotonicity is not available even though the map is smooth.

The diagnostic is therefore two-part. Global injectivity asks about monotonicity and the rank of $\mathbf W$; local geometric stability asks about the tangent-restricted singular values of $\mathbf D_\phi\mathbf W$. Neither question can replace the other.

What the activation does to the metric#

The trouble with everything so far is that you cannot picture it. Rank, kernels, the singular values of a Jacobian in hundreds of dimensions: none of that is an image your intuition can hold. But there is a physical picture that carries all of it. Think of the data manifold as a printed sheet of graph paper and the layer as a hand that deforms it. A smooth, strictly monotone activation stretches the sheet, unevenly, but without tearing or creasing: grid lines stay grid lines. Saturation crushes whole regions of the grid toward a point. And ReLU folds the sheet, pressing everything on the wrong side of the crease flat onto the crease itself. Once folded, two different points of the sheet lie on top of each other, and no later hand can pull them apart again; it can only carry the folded stack along.

Every object in the algebra has a counterpart on the sheet:

And the picture predicts before the algebra confirms: once a direction lies in a layer Jacobian’s null space, later multiplication cannot recover it. The animation then measures how often that exact loss—and the softer decay around it—occurs in the chosen random stack.

With the picture in hand, the equation can name it. The grid spacing the deformed sheet carries is the pullback metric induced by the layer on a submanifold $M \subset \mathbb{R}^d$,

This is the metric the network thinks the data lives in. $\mathbf{D}_\phi$ enters twice, squared, and it does two distinguishable kinds of damage.

Directional rescaling. Each row of $\mathbf{W}$ is weighted by $\phi'(z_i)^2$ in $g_M$. Sigmoid and tanh saturation, the leaky-ReLU $\alpha$ slope, every situation in which a $\phi'(z_i)$ goes small: all push the corresponding row’s contribution toward zero. The “learned distance” the layer imposes is dominated by the rows whose neurons haven’t saturated; the rest contribute almost nothing to perceived similarity.

Directional erasure. When $\phi'(z_i) = 0$ exactly, the row drops from $g_M$ entirely. The metric becomes singular along directions in $\ker \mathbf{J}_F \cap T_\mathbf{x} M$: distances collapse to zero. This is the manifold-side picture of rank collapse, the geometric statement that the layer has stopped being a homeomorphism at $\mathbf{x}$.

The sheet metaphor turns literal in the panel. Identity and GELU stretch the disk; sigmoid and tanh compress it without folding; ReLU folds the negative half-planes onto the axes and crushes entire wedges of the disk onto a 1D set. There is no separate metric tensor the network keeps somewhere, the post-activation grid spacing is the metric.

High dimensions make this worse, not better#

The intuition that “with a wide enough layer, ReLU sparsification is fine” survives in width but not in pressure. Under the simplest model, $\mathbf{z} \in \mathbb{R}^n$ with each $z_i$ independent and symmetric, the probability that at least one coordinate is zeroed is

By $n = 10$ this is $99.9\%$. In a transformer hidden layer of width $768$, every forward pass has approximately half its coordinates zeroed at every point. Whether this turns into rank collapse depends on the structure of $\mathbf{W}_{S(\mathbf{x}), :}$, but the pressure toward sparsification does not disappear in the limit, it becomes the operating regime, and the analysis above stops being worst-case and becomes typical.

High dimensions guarantee that ReLU gating is the operating regime, not that rank collapse is. In a wide layer, half the rows can vanish while the survivors still span every input direction. The quantity to measure is the smallest tangent singular value of the surviving Jacobian, not whether at least one coordinate is zero.

The expressivity–geometry tradeoff#

Why have an activation at all? Without one, a stack of $L$ layers is the single linear map $\mathbf{W}_L \cdots \mathbf{W}_1$, no nonlinear class boundary, no useful expressivity. The activation buys selectivity: $\phi'(z_i) \approx 1$ when a neuron’s prototype matches the input and the projection passes through, $\phi'(z_i) \approx 0$ when it doesn’t. Selectivity is what the activation is for.

Selectivity can damage geometry, but there is no universal one-dimensional tradeoff between the two. A pointwise nonlinearity can be expressive while remaining locally injective, and training can orient $\mathbf W$ so that heavily rescaled rows are redundant. The measurable question is how the task’s tangent directions align with the Jacobian’s singular directions after training.

Reading common tricks as Jacobian regularization#

If pointwise activations really are a standing threat to the Jacobian, you would expect the field to have been fighting them all along, without necessarily saying so. It has. Several standard practices, usually treated as separate phenomena, are all variations on a single intervention: keep $\mathbf{J}_F$ away from rank collapse and saturation.

Residual connections turn each layer into $\mathbf{x} + F(\mathbf{x})$, whose Jacobian is $\mathbf{I} + \mathbf{D}_\phi \mathbf{W}$ instead of $\mathbf{D}_\phi \mathbf{W}$. The identity term does not guarantee full rank: an eigenvalue of the residual branch can still land at $-1$. It does give a clean sufficient condition. If $\|\mathbf D_\phi\mathbf W\|_2<1$, then every singular value of the residual Jacobian is at least $1-\|\mathbf D_\phi\mathbf W\|_2$.

Batch and layer normalization control the scale of the pre-activation $\mathbf z$. For saturating activations this can keep more samples near the high-derivative region; for ReLU, centering a symmetric distribution keeps the active fraction near one half. Neither operation certifies a Jacobian rank or condition number, but both make the derivative distribution easier to control.

Weight and spectral normalization bound the singular values of $\mathbf{W}$. They have no direct effect on $\mathbf{D}_\phi$, but by keeping $\mathbf{W}$‘s spectrum tight they prevent the linear factor from compounding whatever damage $\mathbf{D}_\phi$ has already inflicted.

None of the three replaces the activation. Each controls a different factor in the Jacobian, but only a measurement of the composed, tangent-restricted Jacobian can say whether the representation preserved the geometry a downstream metric will read.

Why this matters for evaluation#

This is where the question from the top comes due. Cosine similarity, $k$-nearest neighbors, clustering, retrieval: every downstream operation on representations assumes the space carries the geometry it is reading. If the layer has collapsed rank, cosine similarity compares vectors whose angles are artifacts of the surviving directions rather than properties of the data manifold. If the layer has saturated, small distances in representation space correspond to entirely different scales of input distance depending on which coordinates were saturated where. The metric you evaluate with is not the metric the network actually exposed.

The conclusion is not “don’t use cosine similarity.” It is that cosine similarity (and every other downstream metric) is only meaningful when the network has preserved the geometric structure the metric is reading. Choose activations that preserve Jacobian rank where the task requires it. Control input magnitudes via normalization so the activation does not saturate. Match the evaluation metric to the geometry the architecture actually preserves. None of this is optional if the goal is to compare representations rather than collateral.

The kernel alternative#

A kernel offers a different primitive. A symmetric positive-semidefinite kernel $K(\mathbf z,\mathbf z')$ provides a feature-space inner product and therefore the induced pseudometric

It becomes a true metric only when the feature map separates points. A finite kernel layer then computes an expansion in kernel sections,

with a closed-form RKHS norm $\|f\|^2_{\mathcal{H}_K} = \boldsymbol{\alpha}^\top \mathbf{K} \boldsymbol{\alpha}$. This does not guarantee that a finite sampled feature map is injective or well conditioned, but it states the comparison geometry explicitly enough to test both properties.

Expressivity never required a pointwise nonlinearity. It required a nonlinearity, and the pointwise form is one choice among several, the one that inserts a diagonal matrix between your data and its geometry and then asks that matrix to do the discriminating. Until you swap the primitive, the diagonal is there on every layer of every forward pass, and it is the thing deciding what your cosine similarities are allowed to mean. So choose $\phi$ with the care you would give a loss function, and check the Jacobian of what you built before you trust a distance computed on top of it.

@misc{bouhsine2026activationsarebadforgeometry,
  author       = {Bouhsine, Taha},
  title        = {What Activations Do to Geometry},
  year         = {2026},
  month        = {feb},
  howpublished = {\url{https://tahabouhsine.com/blog/activations-are-bad-for-geometry/}},
  note         = {Blog post, Records of the !mmortal Data Scientist}
}

@unpublished{bouhsine2026manifoldsactivations,
  author = {Bouhsine, T.},
  title  = {Manifolds, Activations, and Lost Geometry: How Pointwise Nonlinearities Break the Map},
  year   = {2026},
  note   = {Unpublished manuscript}
}