Not All Infinities Are Equal: The Cross-Entropy Asymmetry Behind Hallucination

Show a vision-language model a photo containing five objects and it will happily describe ten. The five inventions arrive with full confidence. The opposite failure, staring at a real object squarely in frame and refusing to mention it, almost never happens. Overgeneration is everywhere; undergeneration is rare. Why would a model trained to match the truth develop such a lopsided way of missing it?

One answer, the one this post develops, is that the loss it was trained with is lopsided in exactly that direction. The suspect is cross-entropy, and specifically its singularity structure. Everyone knows cross-entropy diverges; the standard story stops at “if the supports don’t overlap, the loss is infinite.” The full story is sharper on three counts. The divergence is triggered by a single uncovered coordinate, not by disjoint support, and it comes with a rate: a continuous measure of how much truth the model is missing. The penalty structure is asymmetric, missing truth costs everything while adding falsehood costs, per coordinate, nothing, which is the lopsidedness above in one line. And the boundary where the loss blows up turns out to be exactly the configuration that some of our most popular objectives, the contrastive losses behind CLIP and its relatives, explicitly reach for.

One contract before the mathematics, because this post mixes theorems with interpretation and the two deserve different levels of trust. The statements about the loss itself, where it diverges, how fast, how asymmetrically, are proven facts, and I state them as theorems. The links from those facts to hallucination, to the modality gap in CLIP and SigLIP, and to the extreme batch sizes of InfoNCE-style training are a reading, not a derivation: each phenomenon has competing explanations in the literature, and the singularity is one factor among them, not a sole cause. What the reading buys is a single mathematical reference point for three failure modes that are usually discussed separately. That division of trust holds everywhere below, and I will restate it at the end.

The singularity is not binary#

Assumptions for the theorems below. Finite alphabet $\mathcal{X}$, discrete distributions $p, q$, natural logarithm, and the standard extended-real convention $p_i \log 0 = -\infty$ for $p_i > 0$ (so $0 \log 0 = 0$). Under these assumptions the support condition below is exact. In continuous or countably-infinite settings KL and cross-entropy can be infinite for additional integrability and tail reasons even when no single coordinate has $q_i = 0$; those cases are out of scope here.

The textbook condition for $H(p, q) = +\infty$ is “if the supports are disjoint.” The actual condition is strictly weaker. Define the violating set

and the violating mass $S(p, q) = \sum_{i \in V_{p,q}} p_i$. Then

A single uncovered coordinate is sufficient. Two distributions $p = (0.5, 0.3, 0.2)$ and $q = (0.7, 0.3, 0)$ overlap on 80% of $p$‘s mass and still give $H(p, q) = +\infty$. The textbook framing of “disjoint support” is presenting a sufficient condition as if it were necessary and sufficient.

But the binary infinite/finite picture is itself misleading. The path to infinity is continuous. If we smooth $q$ uniformly, $q^{(\varepsilon)} = (1 - \varepsilon) q + \varepsilon u$ with $u$ uniform on $|\mathcal{X}| = n$ symbols, then as $\varepsilon \to 0^+$,

The slope of the divergence in $-\log \varepsilon$ is exactly the violating mass $S(p, q)$. The “infinity” of cross-entropy is not a single rate, it is a real-valued function of how much truth $q$ is missing.

A model that fails to assign mass to a coordinate $i$ where $p_i = 0.01$ is, in the limit, infinitely wrong. A model that fails on a coordinate where $p_i = 0.9$ is ninety times more wrong, in the precise sense that its loss diverges ninety times faster as you smooth toward zero; a model that misses everything, $S = 1$, diverges a clean hundred times faster than the 1% model, the panel’s headline case. The textbook view collapses all of these together into “the loss is infinite.” The view that lets you reason about which model is worse, and by how much, distinguishes them by $S(p, q)$.

Missing truth is infinite. Adding falsehood is free.#

The violating mass grades how badly a model misses the truth. But missing truth has a mirror image, putting mass where the truth has none, and on that side the loss stops being even-handed. Cross-entropy is not symmetric, in general $H(p, q) \neq H(q, p)$, and the asymmetry has a precise consequence at the boundary of the simplex. For each coordinate:

The first case is the model failing to cover a real outcome. The second is the model assigning probability to something that cannot happen. Cross-entropy treats these radically differently: the first is unbounded, the second is exactly zero.

The per-coordinate reading is sharp. In isolation, putting mass on a real outcome the model had assigned zero to is the only way to incur an unbounded penalty, and putting mass on an outcome that cannot happen is, term by term, free.

So is adding falsehood literally free? Per coordinate, yes, exactly. Per training step, not quite, and the gap is the simplex constraint: in a softmax output, mass placed on a falsehood necessarily comes from somewhere, including, in general, from real outcomes, so falsehood competes with coverage through the normalization. The asymmetry is therefore a bias, not an unconstrained optimum, but the bias is real and it points unambiguously toward overcoverage under uncertainty.

The gradient makes the two halves explicit. For softmax cross-entropy with logits $z$, $\partial H / \partial z_j = q_j - p_j$. On a target-zero coordinate ($p_j = 0$) the gradient is exactly $q_j$, which actively pushes false mass down, finite, normalization-mediated pressure, not zero. The per-coordinate “free” statement is about the direct loss term $-p_j \log q_j$, which vanishes; the indirect penalty, routed through normalization, is real but bounded. And because the softmax never outputs an exact zero from finite logits, $q_i = 0$ is a boundary limit the loss reaches toward, not a state training actually attains. The infinity is a property of the boundary the gradient flows toward, not a value the loss usually takes.

With that caveat in place, the bias still does most of the explanatory work, and it is the mechanism behind the photo this post opened on. The model describing ten objects in a five-object image pays a far smaller marginal cross-entropy for distributing some of its probability over the five hallucinated descriptions than it would pay for zeroing out any of the five real ones. The per-coordinate calculus shows up at every training step, in every gradient, and is consistent with the overgenerate-rather-than-undergenerate pattern observed in instruction-tuned models. Pre-training spent its gradients on coverage; refusal, declining to assign probability mass to a wrong answer, looks like the behaviour the loss had been steadily discouraging.

Cross-entropy diverges at disjoint support#

Both the asymmetry above and the growth-rate story reduce, in the worst case, to the same singularity. The Kullback–Leibler divergence

inherits the singularity directly. $H(p)$ is finite for any distribution on a finite alphabet, so $D_\text{KL}(p \,\|\, q) = +\infty$ exactly when $H(p, q)$ is.

The case of disjoint support is the cleanest example, and the most consequential. Disjoint support is the probabilistic analog of vector orthogonality, two distributions that share no events are like two vectors that share no projection. (Analog of orthogonality, not of statistical independence: a factorized joint $P_{XY} = P_X P_Y$ has KL-to-product equal to the mutual information, which is zero, not infinite. The boundary that breaks the divergence is the failure of one distribution to dominate the other.) But the two settings handle this configuration in incompatible ways.

This is not a notational accident. Cross-entropy and KL were designed to be finite divergences, useful as proxies for “how far apart are these distributions.” But they stay finite only for distributions in the interior of the simplex, distributions with full support. At the boundary of the simplex, where one distribution’s mass goes to zero on a coordinate the other still covers, the divergence runs to $+\infty$ under the standard extended-real convention; it does not stay finite there.

Contrastive losses ignore this. InfoNCE, SimCLR’s loss, CLIP’s loss, and supervised contrastive learning all want negative pairs to be orthogonal in representation space, they all reach for the boundary as their objective. Each one is built on a divergence that blows up there.

The modality gap as a singularity shadow#

What happens to a training run that keeps reaching for the configuration its own loss diverges at? Multimodal contrastive learning is the clearest place to look. CLIP and SigLIP train image and text encoders to produce embeddings in a shared unit sphere; the contrastive objective wants matched image-text pairs close together and mismatched pairs far apart, with “far” operationalised as small (eventually negative) cosine similarity. The cleanest place to apply the singularity lens is the per-anchor softmax: for each anchor, the loss pressures its one positive cross-modal partner to dominate every mismatched cross-modal sample, driving the negative-pair similarities toward the floor. It is that per-anchor negative distribution, not the global marginals, that the loss pushes toward a boundary configuration. This is deliberately not the stronger claim that the image and text marginal supports must be disjoint, $\mathrm{supp}(p_\text{img}) \cap \mathrm{supp}(p_\text{txt}) = \varnothing$: if anything, well-aligned positives make the cross-modal marginals overlap more, not less. The pressure lives in each anchor’s negatives.

But the loss diverges at that boundary. The optimisation cannot reach full orthogonal separation of the negatives; the gradient blows up as the per-anchor objective approaches it. So it settles for near-orthogonal clusters separated by a residual distance. The well-documented “modality gap” in CLIP and SigLIP, the persistent shift between image and text clusters that no amount of training closes, is consistent with the optimisation keeping a safe distance from a singularity at exactly the configuration the loss is asking it to reach.

This is the interpretive side of the contract, and the modality gap is where the competing explanations are strongest. The literature has at least three that any complete account has to engage with. Liang et al.\ (2022) showed the gap exists at initialisation, before any contrastive training; that part of it is a property of random initial encoders, not of the loss. Deep-net cone effects produce embeddings concentrated in narrow regions of the sphere irrespective of contrastive objectives. Optimisation dynamics around the temperature parameter shape how aggressively the loss pushes negatives across the equator. The singularity shadow is one factor among these, not the sole cause.

The batch-size story sits under the same contract. InfoNCE’s gradient on a negative pair is weighted by its softmax probability, and in high dimensions random unit vectors concentrate at cosine zero with variance $1/d$. The negatives the loss wants to push away are mostly already near orthogonality, so the gradient on each one is small. Accumulating useful gradient from a flat region is consistent with the regime that demands enormous batches. The catch is that this is one factor among several known to make large batches help contrastive learning: hard-negative mining (large batches see more genuinely hard negatives), gradient variance reduction (large batches give cleaner stochastic gradients), and architectural effects on the temperature schedule all contribute. CLIP’s 32,768-pair batches and the multi-gigabyte similarity matrices are the result of these factors composing; the singularity-flatness reading explains some of that pressure, not all of it.

What the singularity actually says#

So where does this leave anyone training with these losses? The proven core is compact. Cross-entropy is a finite-distribution divergence with three structural properties:

1. It diverges as soon as $q$ leaves any of $p$‘s support uncovered, at a rate determined by the missing mass.
2. It treats coverage failures and precision failures infinitely asymmetrically.
3. It runs to $+\infty$ at the boundary, disjoint support, the configuration the contrastive objective reaches for, rather than taking a finite value there the way a true metric would.

A practitioner can do one of three things with this. Live with it, and apply the standard mitigations: label smoothing pulls $q$ off the simplex boundary, temperature scaling slows the gradient near the singularity, support regularization explicitly penalizes the violating mass. Avoid the boundary, and design objectives that don’t ask for what cross-entropy cannot deliver, SigLIP’s bias parameter is exactly this move, since it requires only that mismatched similarities sit below some threshold, not at the limit. Change the divergence: Jensen–Shannon is bounded by $\log 2$ everywhere; Wasserstein measures distance under transport rather than overlap; MMD operates in an RKHS and is finite for any pair of distributions. Each removes the singularity at a different structural cost.

The contract from the top, restated now that the pieces are on the table. The proven part of the argument is small and load-bearing: the singularity exists, it is weaker than disjoint support, its severity is a continuous function of violating mass, and its asymmetric structure is a precise per-coordinate fact. The interpretive part, the link from these properties to hallucination, the modality gap, and contrastive batch sizes, is a unifying lens, not a derivation. The lens is, I think, a useful one: it puts three failure modes that are usually discussed separately under a single mathematical structure. And it makes at least one falsifiable call: if the extreme batches of InfoNCE-style training are partly payment for pushing negatives toward a boundary the loss diverges at, then an objective that stops short of that boundary should train well without them. SigLIP’s thresholded sigmoid is that objective, and its strong small-batch results are consistent with the call, though consistent-with is not proof-of. Each of these phenomena still has competing explanations, and a complete account of any of them has to do more than point at the singularity.

What is not an option is pretending the singularity isn’t there. The choice is not whether the loss has a singularity. The choice is whether one knows where it is, and how much of the model’s behaviour one is willing to read through that lens.

@misc{bouhsine2026notallinfinitiesareequal,
  author       = {Bouhsine, Taha},
  title        = {Not All Infinities Are Equal},
  year         = {2026},
  month        = {feb},
  howpublished = {\url{https://tahabouhsine.com/blog/not-all-infinities-are-equal/}},
  note         = {Blog post, Records of the !mmortal Data Scientist}
}

@unpublished{bouhsine2026onthe,
  author = {Bouhsine, T.},
  title  = {On the Singularity Structure of Information-Theoretic Losses},
  year   = {2026},
  note   = {Unpublished manuscript}
}