Softmax in Java: why harmless-looking numbers produce NaNs
Fix overflowing softmax in an editable Java notebook, then discover why stable cross-entropy must work directly from logits.
Run cells from top to bottom. Typed declarations and imports carry into later cells. Shift + Enter runs a cell. Reloading or leaving this page discards edits and Java state.
Add 1,000 to every score in a three-class classifier. The ordering stays the same. The gaps stay the same. The softmax probabilities should stay the same, too. A direct Java implementation returns three NaNs instead.
Imagine a classifier choosing between Cat, Dog, and Bird. It produces one raw score, or logit, for each class. Softmax converts those scores into probabilities:
| Array index | Class | Logit (input score) | Softmax probability (output) |
|---|---|---|---|
| 0 | Cat | 2.5 | 0.546870 ≈ 54.7% |
| 1 | Dog | −0.5 | 0.027227 ≈ 2.7% |
| 2 | Bird | 2.25 | 0.425903 ≈ 42.6% |
The position in the array identifies the class; the value is that class's score. Index 0 means Cat, but Cat's score is 2.5. Scores can be negative and need not sum to one. The output probabilities keep the same class order and sum to one.
A label identifies the correct class for an example. If the picture actually shows a dog, its label is index 1, regardless of the scores. Here the model predicts Cat, the class with the highest probability, so that prediction would be wrong.
Now add 1,000 to every score: [2.5, -0.5, 2.25] becomes [1002.5, 999.5, 1002.25]. The relative preferences and desired probabilities stay the same. Yet exponentiating the second set overflows even Java's double. Dividing infinity by infinity does not recover the probability we wanted.
These are constructed inputs, chosen to expose the arithmetic. No model training or dataset is needed. Run the cells in order: later cells reuse earlier definitions. The notebook uses CPU Java in the browser's emulated JVM, with a double-precision reference followed by an F32 jax4j calculation. The first run includes loading the runtime.
Make the obvious implementation fail
Softmax first turns logits into positive weights by exponentiating each score. Dividing each weight by their sum gives the probabilities. The output can be used as a class probability distribution, although that alone does not establish that a model's confidence is calibrated.
For scores z, the formula is p[i] = exp(z[i]) / sum(exp(z[j])). Java translates it almost word for word. The following deliberately naive function has no input validation; we will give the stable version a clearer contract.
1. Overflow in three numbers
The output probabilities are approximately [0.546870, 0.027227, 0.425903]. The naive shifted probabilities are [NaN, NaN, NaN]. A double has a large range, but Math.exp(1000) exceeds it. Switching from float to double merely moves the boundary.
Negative scores expose the other end of the range. Try [-997.5, -1000.5, -997.75]: all three exponentials round to zero in double precision, leaving zero divided by zero. The desired probabilities are still the same as before.
Move the largest score to zero
Subtract a common value m from every logit. Each exponential acquires the same factor exp(-m), which cancels between numerator and denominator. In exact arithmetic, softmax is unchanged.
Choose m as the maximum score. Now every shifted score is nonpositive, so every exponential is at most one. At least one is exactly one. For our inputs, [1002.5, 999.5, 1002.25] becomes [0, -3, -0.25], and the denominator is about 1.828588.
This function accepts a nonempty array of finite scores. It rejects NaNs and infinities rather than assigning them an accidental meaning. Its output is a fresh array; the input remains available for comparison.
2. Stable softmax in plain Java
For these exactly representable quarter-step inputs, the shifted differences are identical, giving zero measured shift error and a sum of one to the printed precision. In general, compare with a tolerance: adding a large constant can itself round away small differences before softmax sees them. No normalization formula can reconstruct those lost bits.
Max-shifting prevents exponential overflow for finite inputs, but it does not prevent every underflow. An extremely negative shifted score can still exponentiate to zero. For reporting a very unlikely class, that may be an acceptable approximation. For calculating its training loss, it creates another problem.
A zero probability need not mean an infinite loss
Suppose the scores are [0, -1000, -1001], in the same Cat, Dog, Bird order. The correct label is Dog, class index 1. Its score is −1000; the number 1 selects its position in the array. The classifier strongly prefers the wrong answer. Cross-entropy for this one example is the negative natural logarithm of the probability assigned to the correct class.
Our stable softmax returns [1, 0, 0] in double precision: the last two exponentials underflow. Consequently, -Math.log(p[1]) returns infinity. The mathematical loss is finite, approximately 1000. We can preserve it by staying in logarithms.
Write the loss as log(sum(exp(z[j]))) - z[target]. The logarithm of that exponential sum is called log-sum-exp. Apply the same maximum shift, and arrange the loss as (m - z[target]) + log(sum(exp(z[j] - m))). Keeping the large score difference separate also avoids adding a small logarithm to a large maximum only to subtract it again.
3. Cross-entropy directly from logits
The outputs are Infinity, 1000.0, and 0.853544003. The last value uses Bird (index 2) as the label for our original scores, so it is the negative logarithm of Bird's approximately 42.6% probability. The second result is rounded: mathematically the loss is very slightly above 1000. Underflow in the tiny denominator contributions no longer destroys the large wrong-class penalty.
Clamping probabilities to an arbitrary floor would cap that penalty and change the objective. Computing from logits avoids introducing such a floor. This remains floating-point arithmetic, though: scores near opposite extremes of the double range can overflow even their difference. Our small example does not claim arbitrary-range accuracy.
Normalize each row with jax4j
A batch of examples is usually a matrix: rows are examples, and columns are Cat, Dog, and Bird in that order. Each row needs its own maximum and denominator. A global maximum may leave another row entirely underflowed; a global sum mixes separate probability distributions.
The next cell spells out the operations with jax4j. max(1, true) reduces the class axis while retaining a one-column shape. Broadcasting subtracts each row's maximum from that row's three scores. The same shape rule makes the denominator divide each row separately. The broadcasting lesson explores that alignment in more detail.
4. Two independent distributions
Expect flattened probabilities close to [0.546870, 0.027227, 0.425903, 1, 0, 0] and two row sums close to one. The tensor calculation uses F32, so compare against the double reference within roughly 1e-6 for these inputs. Jit.compile captures a reusable tensor expression; this tiny CPU example makes no performance claim. Traced intermediates are managed by the compiled expression, while the cell explicitly closes its input and output tensors.
This is the numerical issue underneath a classifier such as our MNIST Java notebook. For parameter derivatives and training objectives, continue with a loss function and its parameter gradients. Here the narrower question is whether the arithmetic preserves the distribution and loss we meant to calculate.
Three useful checks before training
Keep the failure cases when replacing a naive implementation. Ordinary inputs alone are too polite to expose this bug.
| Input logits | Check | Reference |
|---|---|---|
[2.5, -0.5, 2.25] and [1002.5, 999.5, 1002.25] | Same probabilities within tolerance | [0.546870, 0.027227, 0.425903] |
[7, 7, 7] | Equal scores split the mass equally | Each probability is one third |
[0, -1000, -1001], target 1 | Loss from logits remains finite | Approximately 1000 |
Also check each row separately for nonnegative finite entries and a sum near one. A flattened batch can sum to its row count while individual rows are wrong. Reject empty inputs and decide explicitly how your application handles nonfinite scores.
For a deeper treatment of rounding error, see Blanchard, Higham, and Higham's Accurate Computation of the Log-Sum-Exp and Softmax Functions. Their analysis explains why algebraically equivalent formulations can behave differently in finite precision.
Try changing the second cell's common shift to a negative value, then change a score gap instead. A common shift should preserve the distribution within rounding error. A changed gap should alter it. Those two edits make a useful boundary between a numerical implementation problem and an actual change in what the classifier predicts.
