Cache the dice, not the price: common random numbers as a memo table

Here is the experiment, and then the trap that came with it.

Take the one-step European tape from the previous article, compiled to bytecode, running on eight threads. Price it at seven spots — 98, 99, 99.5, 100, 100.5, 101, 102 — with the same seed and the same twenty million paths every time. That is a delta ladder, and it is the shape of most of what a risk system does all day: the same paths, again, under a slightly different market.

Plain cpu-jit does the ladder at 155.6 million paths per second per call. Turn on one enum constant, JitOptimizations.Category.DRAW_CACHE, and calls two through seven run at 285.4. Same seed, same paths, same seventeen digits of price at every spot. 1.83× for typing a name.

Now change one number that is not in the maths, not in the market and not in the kernel: the path count. From 100 million to 150 million. The speedup goes from 1.89× to 1.01×. Nothing throws, nothing logs, the price is still right, and the cache you switched on has silently switched itself back off.

That pair of results is the whole article. What the draws depend on and what they don't; why reusing them is the statistically right thing before it is the fast thing; what a memo table over a Philox stream looks like in two engines; how much it is worth on four combinations of engine and pass; and why the number that decides whether you get any of it is a memory cap you have to know exists.

Every number below was measured on referential machine (Ryzen 7 8845HS, 8 cores, Zulu OpenJDK 25.0.1) for this article, as the median of three seven-point spot ladders, one fresh JVM per row.

Part one: what a draw depends on

Go back to the tape. Twenty-six nodes, and the one that matters here is number 14:

 14  RANDN  v14  = randn #0                      -

Every other node is a function of the nodes above it and, ultimately, of the five inputs — spot, strike, vol, rate, maturity — that arrive in in[]. Node 14 is not. It is the only node that reads from outside the tape, and what it reads is filled in by the replay loop before the kernel is called:

fillDraws(draws, path, seed);          // Philox(path, seed) → Box-Muller → draws[]
kernel64.forward(v, in, draws, scratch);

fillDraws takes the path index and the seed. It does not take in. A Philox generator is a pure function of its counter and key, so the normal that lands in draws[0] for path 40,000,000 under seed 42 is the same number whether spot is 98 or 102, whether vol is 20% or 80%, whether you are pricing a call or a digital.

Which means a delta ladder — seven markets, one seed, one path count — is asking the RNG the identical question seven times and getting the identical answer seven times, at 38% of the cost of a price path each time. A previous measurement on this box put Philox plus Box-Muller at 15.9 ns of a 42.3 ns single-threaded path. That is a third of the bill for computing something you already have.

Part two: it was the right thing to do anyway

Before it is an optimisation, reusing the draws across a ladder is a statistics decision, and an old one. It has a name — common random numbers — and it is in every simulation textbook because of a one-line identity:

Var(A − B)  =  Var(A) + Var(B) − 2·Cov(A, B)

A delta ladder is a collection of differences. Delta is (V(100.5) − V(99.5)) / 1, a curvature charge is V(up) − V(base), a VaR shock is V(shocked) − V(base). Simulate the two legs on independent draws and the covariance is zero: the noise of the difference is the noise of both legs added together, and each leg's noise is the full Monte-Carlo standard error of a price. Simulate them on the same draws and the two prices move together — path 7 finishes in the money under both markets, path 8 out of the money under both — so the covariance is nearly all of the variance and the difference is almost quiet.

Measured, because this blog does not quote textbook identities without checking they still hold on a Friday. Twenty independent repetitions of a central-difference delta on the European, two million paths per leg, bump ±0.5:

how the two legs were simulated	mean delta	standard deviation across 20 runs
same seed for both legs (CRN)	0.598805	0.000552
a fresh seed for each leg	0.599234	0.017428
adjoint, one sweep, for reference	0.598723	—

Thirty-two times less noise from the same number of paths, which is a thousand-fold variance reduction, for free. Nobody sensible runs a bump ladder any other way, and NablaTensor's own bump cross-check and FRTB curvature showcase both run their bump legs on common random numbers and say so.

So a revaluation harness was always going to pass the same seed and the same path count to every call. DRAW_CACHE is what happens when the engine notices.

Part three: a memo table over a stateless generator

Here is the whole mechanism in cpu-jit, from JitReplay.java:

boolean useCrn = crn && randCount > 0 && (long) paths * randCount <= CRN_CAP;
if (useCrn) {
  boolean hit = cacheFilled && seed == cacheSeed
      && pathOffset == cachePathOffset && paths == cachePaths;
  if (hit) {
    cacheRead = true;
  } else {
    int need = Math.toIntExact(paths * randCount);
    if (cache64 == null || cache64.length < need) cache64 = new double[need];
    cacheSeed = seed; cachePathOffset = pathOffset; cachePaths = paths;
    cacheFilled = false;
    cacheWrite = true;
  }
}

That is a memo table with exactly one entry. The key is the triple (seed, pathOffset, paths); the value is every normal draw of the last replay, paths × randCount doubles, laid out path-major so that each path's draws are contiguous. A hit means the triple matches exactly. A miss means the block is (re)generated and written through. There is no eviction policy because there is nothing to evict: the next different triple simply overwrites.

Then, in the path loop, the only change is where draws[] comes from:

if (read) {
  System.arraycopy(cache64, slot, draws, 0, randCount);   // the cache
} else {
  fillDraws(draws, path, seed);                            // Philox + Box-Muller
  if (write) System.arraycopy(draws, 0, cache64, slot, randCount);
}
kernel64.forward(v, in, draws, scratch);

Nothing about the generated kernel changes. It still receives a draws[] array; it has no idea whether the numbers in it came from ten rounds of Philox or from a memcpy. That is the property that makes this safe: the compiled tape was already agnostic about where its randomness came from, so the optimisation cannot change the answer even in principle.

The simd engine has its own copy in BatchedReplay.java, same key, same semantics, one layout difference that follows from how the vectorised engine works. It evaluates thirty-two paths at a time, node by node, so when it reaches a RANDN it wants draw k for all thirty-two paths at once. Its cache is therefore draw-major:

/** index of draw k for the batch starting at global path base. */
int index(long base, int k) {
  return k * padded + (int) (base - origin);
}

Every RANDN node becomes one arraycopy of thirty-two contiguous doubles into the batch's value row. Same idea, transposed, because the consumer is transposed.

Two switches

One difference between the engines is not about layout and will bite you. cpu-jit decides per kernel:

this.crn = "on".equals(System.getProperty("nablatensor.crn"))
    || jit.has(Category.DRAW_CACHE);

so .jit(JitOptimizations.Category.DRAW_CACHE) on the model builder is enough, and two pricers in one JVM can disagree about it. simd decides once, at class-load time:

private static final boolean CRN = "on".equals(System.getProperty("nablatensor.crn"));

and never consults the fluent options. If you name DRAW_CACHE on a simd model and forget -Dnablatensor.crn=on on the command line, you get the plain engine and no message. Every simd number in this article was measured with the system property set.

Part four: what it is worth

The methodology, because it matters more than usual here: build one kernel, warm HotSpot on a different seed so the ladder's first call is a genuine miss, then issue seven calls with only spot changed. "Cold" is the first of the seven — it pays for the RNG and for writing the block. "Warm" is the mean of the remaining six. Three such ladders, a fresh seed each, median reported. The "plain" column is the same ladder with the cache off, which is the honest baseline: a call that neither reads nor writes a cache.

One draw per path: the one-step European

Twenty million paths, fp64, eight threads. The cache block is 160 MB.

engine	pass	plain	cold (writes)	warm (reads)	warm vs plain
cpu-jit	price only	155.6 Mpath/s	151.8	285.4	1.83×
cpu-jit	value + 5 Greeks	98.9	100.2	150.1	1.52×
simd	price only	340.9	364.4	632.1	1.85×
simd	value + 5 Greeks	250.7	220.2	340.9	1.36×

Two things to read off. First, the price pass gains the same 1.8× on both engines, and that number is a measurement of something real: if removing the RNG makes a path 1.83× faster, the RNG was 1 − 1/1.83 = 45% of the path. The earlier single-threaded microbenchmark said 38%. The ladder is the more trustworthy of the two — it measures the generator in situ, at eight threads, with everything else in the loop competing for the same core — and the honest reading is that the standalone-loop figure was a slight underestimate, by about the amount you would expect when the transcendental calls in Box-Muller stop overlapping with the tape's own exp.

Second, the Greeks pass gains less, on both engines, and simd least of all. That is Amdahl's law, not a defect. The reverse sweep never touches the RNG — look at the reverse rules in the previous article: RANDN emits nothing — so adding it to the path dilutes the RNG's share. cpu-jit's adjoint sweep costs 1.66× a price pass and its share falls from 45% to 34%; simd's costs 1.96× and the share falls to 26%. The cache removes the same nanoseconds from both; the Greeks path was just longer to begin with.

252 draws per path: the Asian

Three hundred thousand paths of the 252-step arithmetic Asian — 1,536 tape nodes, 252 normals per path. The cache block is 300,000 × 252 × 8 bytes = 605 MB, comfortably under the cap, which is why the path count is what it is.

engine	pass	plain	cold (writes)	warm (reads)	warm vs plain
cpu-jit	price only	1.69 Mpath/s	1.69	4.62	2.73×
cpu-jit	value + 5 Greeks	1.44	1.50	3.07	2.13×
simd	price only	4.29	3.81	7.69	1.79×
simd	value + 5 Greeks	3.12	2.90	3.97	1.27×

Now the RNG is 252 Philox calls and 126 Box-Muller pairs per path against a 1,536-node tape, and on cpu-jit it is the biggest single line item: 63% of a price path, 53% of a Greeks path. 2.7× on the price ladder, from an optimisation that emits no code and changes no arithmetic.

simd gains less here — 1.79× and 1.27× — and this time the reason is not dilution alone. Its Philox is vectorised, thirty-two counters through the rounds in parallel, so the RNG is a smaller share of a simd path to begin with: 44% of the price pass, 21% of the Greeks pass. The engine that had already made the draw cheap has less to save by not doing it. The cache rewards whichever engine was worst at generating random numbers, which is a slightly humbling way to think about a speedup.

The cold call

Read the "cold" column against "plain" in both tables: −2%, +1%, +7%, −12%, 0%, +4%, −11%, −7%. The first call of a ladder does everything the plain engine does plus one arraycopy per path into a large array, and the cost of that is within the run-to-run noise of these measurements. A pricer with the cache on that only ever makes one call is not slower. It is just not faster, and it is holding a few hundred megabytes for no reason.

Part five: it computes the same number

The cache is a claim that the draws do not depend on the market. If the claim is wrong — some node somewhere folded a market input into the RNG stream — the prices would differ. So, the Asian at spot 100, seventeen significant digits, with the cache off and then on:

252-step Asian, 300,000 paths, 8 threads, fp64
engine   crn   price                  delta
cpu-jit  off   5.3010210548891300     0.56163041913700040
cpu-jit  on    5.3010210548891300     0.56163041913700040
simd     off   5.3010210548890870     0.56163041913700080
simd     on    5.3010210548890870     0.56163041913700080

Identical, all 53 bits of mantissa, on both engines, on both the price and the adjoint delta. The two engines differ from each other in the fourteenth digit — that is simd's thirty-two-wide batch reduction against cpu-jit's per-thread running sum, the same summation-order effect the previous article documented between one and eight threads — but neither engine moves when the cache is switched on. Which is what "the draws never depended on the market" looks like when you check it.

This is also what makes the cache safe to assert on. The engine's tests can demand == between a cached and an uncached ladder, not < tolerance, and any future change that leaks a market input into the draw stream fails the test on the first path.

Part six: the cap, and the silence

Now the trap.

private static final int CRN_CAP = Integer.getInteger("nablatensor.crn.cap", 1 << 27);
// ...
boolean useCrn = crn && randCount > 0 && (long) paths * randCount <= CRN_CAP;

1 << 27 is 134,217,728 elements. In double that is exactly 1 GiB, and it is a sensible default: a cache that can silently allocate several gigabytes because someone asked for a billion paths is not a feature. But look at what happens when the block is one element too large: useCrn is false, the cache is skipped, the engine regenerates every draw on every call, and nothing else changes. No exception. No log line, even with -Dnablatensor.jit.debug=1, which prints crn=true at kernel build time and says nothing at replay time about whether the cache engaged. The price is right. The throughput is simply the plain engine's.

Measured, cpu-jit, price pass, the European, same three-ladder median:

paths	cache block	under the cap?	plain	cold	warm	warm vs cold
100,000,000	800 MB	yes	168.7 Mpath/s	153.3	290.0	1.89×
150,000,000	1.2 GB	no	170.6	158.8	161.1	1.01×
150,000,000, -Dnablatensor.crn.cap=200000000	1.2 GB	yes	—	153.5	274.4	1.79×

Row two is the failure mode in a single line. Everything the caller can see — the flag, the seed, the prices — says the cache is on. The only evidence it is not is a throughput number that looks exactly like the number you would have got without asking for anything, and you will only notice that if you already know what to expect.

The arithmetic you need before you trust a ladder speedup, then: paths × draws per path × 8 bytes. Under 1 GiB, the cache is live. Over it, either raise -Dnablatensor.crn.cap — it is an int count of elements, so it tops out at 2³¹ − 1, or 16 GiB of double — or split the ladder into pathOffset chunks that fit, each of which gets its own hit. (fp32 does not help here: the cpu-jit cache becomes a float[] and halves the memory, but the cap counts elements, so it admits exactly the same number of paths.)

Part seven: what a miss looks like

The key is an exact match on three numbers, and it is worth being concrete about what that excludes, because the failures are all silent too.

• A different seed per call. A harness that re-seeds every valuation gets nothing, and pays the write-through every time. It also loses the variance reduction, so it was already doing the wrong thing.
• A different path count per call. A ladder that prices the base case with 2 million paths and the shocks with 1 million is seven misses.
• A different pathOffset. Chunked replays are fine as long as the chunks are the same chunks each time; a harness that rebalances its chunk boundaries between calls will miss on every chunk that moved.
• Two pricers. The cache belongs to one JitReplay — one kernel, one model. A call and a put priced on the same paths hold two copies of the same block.
• Another engine. The scalar cpu interpreter and every GPU backend (vulkan, cuda, rocm, opencl) have no draw cache in their source as of this writing and regenerate every call. On a GPU that is probably the right trade — thousands of Philox lanes per cycle against a 605 MB block that would have to live in device memory — but it means a ladder that is 2.7× on cpu-jit is 1.0× when you move it.

None of these are bugs. They are the definition of a one-entry memo table. But every one of them presents as "the speedup is smaller than the blog said", and the diagnosis is always the same: print the triple on every call and look for the one that changed.

What to actually use

Everything on one row of the table, plain against cached, both engines pinned to eight threads, warm-ladder throughput:

tape	pass	cpu-jit	cpu-jit + cache	simd	simd + cache
European, 1 draw	price only	155.6	285.4	340.9	632.1 Mpath/s
European, 1 draw	value + 5 Greeks	98.9	150.1	250.7	340.9
Asian, 252 draws	price only	1.69	4.62	4.29	7.69
Asian, 252 draws	value + 5 Greeks	1.44	3.07	3.12	3.97

Turn it on when the same (seed, pathOffset, paths) block is about to be replayed more than once and the block fits in memory. That describes a delta ladder, a VaR shock set, a curvature up/down pair, a scenario grid — the revaluation loop of a risk system, which is the loop that costs the money. Turn it on by naming it:

var model = Nabla.model(EquityMarket.atmOneYear(),
        (rec, in) -> Products.asianCall().record(rec, in, TimeGrid.uniform(252)))
    .greeks().fp64().threads(8).on("cpu-jit")
    .jit(JitOptimizations.Category.DRAW_CACHE)     // the category, not Level.ALL
    .build();

or, for either engine, -Dnablatensor.crn=on on the JVM command line — which on simd is the only switch that works.

Do not turn it on through JitOptimizations.Level.ALL. ALL bundles it with FAST_MATH, which the previous article measured as slower on this CPU and which changes the low bits, so you would be trading bit-exactness for a speedup and then losing part of the speedup. Name the category.

Leave it off for a single valuation, because it cannot help and it holds the block. And whichever way you set it, do the multiplication first: if paths × draws is over 134 million, you are not getting a cache, you are getting a flag that says you have one.

The ranking of what mattered, in the order it mattered:

1. Common random numbers at all. A thousand-fold variance reduction on every difference a ladder computes. Free, old, and the reason the cache has a key to match on.
2. Not generating the same numbers seven times. 1.8× on a one-draw tape, 2.7× on a 252-draw tape, on the pass that runs the most.
3. Knowing about the cap. The difference between the two numbers above and 1.0×, decided by a threshold that does not announce itself.
4. The engine's own RNG speed. simd gains less because it had less to lose. An optimisation that removes work is worth exactly the work it removes, and no engine gets to count the same nanoseconds twice.

Try it

The ladder harness is a few lines against the public API — one kernel, warm it on a different seed, then seven calls with a shocked spot and the seed held fixed:

try (var mc = MonteCarlo.of(Products.europeanCall())
        .market(base).steps(1).fp64().threads(8).priceOnly().on("cpu-jit").build()) {
  mc.run(base, paths, warmupSeed);
  for (double spot : new double[] {98, 99, 99.5, 100, 100.5, 101, 102}) {
    var r = mc.run(base.withSpot(spot), paths, seed);      // same seed, same paths
    System.out.printf("%.1f  %.1f Mpath/s%n", spot, r.scenariosPerSecond() / 1e6);
  }
}

Run it twice, once plain and once with -Dnablatensor.crn=on, and the second run's calls two through seven should be 1.8× the first's. If they are not, multiply paths by the tape's draw count and compare it with 134,217,728 before you look anywhere else. Then set -Dnablatensor.crn.cap higher, or paths lower, and watch the ratio come back.

Which is the moral, and it is the same one as last time from the other side. The previous article's lesson was that when you generate code you are writing to another compiler with budgets you did not choose. This one's is that when you cache, you are writing to a memory budget you did choose — and chose correctly — and it will still go quiet on you at the boundary unless you make it speak. The fix is one println. The measurement is the only way to find out you needed it.

Source: JitReplay.java (the cpu-jit cache and the path loop), BatchedReplay.java and VectorReplayF64.java (the simd cache and its draw-major layout), JitPhilox.java (what is being cached) and JitOptimizations.java (DRAW_CACHE, and why it is not in LOW_RISK).