In Three boring Claude features I walked through an anomaly detector that runs in the background of Split Test Pro: once a day per active experiment, it sends Claude the last 21 days of daily conversion data and asks it to flag four patterns (novelty effect, trend divergence, day-of-week, sudden shift). It works. It’s been in production for months. And it’s built on a mistake I want to write down, because I keep seeing other people make the same one.
The mistake is this: the feature is called anomaly detection, and I let the name talk me into asking a language model to detect anomalies. It doesn’t detect them. It reads a block of numbers and does its best to notice which ones are unusual, in prose, non-deterministically, one API call at a time. That is a statistics problem wearing an LLM costume.
Two jobs, one prompt
If you look closely at what “did anything weird happen last night” actually asks for, there are two separate jobs inside it.
The first is detection: given this variant’s conversion series, is any point unusual relative to the distribution the rest of the series implies? Did the rate shift on a specific date by more than noise? Is there a weekly cycle? Is the lift decaying? These questions have right answers. Two people looking at the same series and the same definition of “unusual” should agree. That’s the tell that it’s a statistics problem, not a judgment one.
The second is interpretation: this point is unusual — so what? Is a Tuesday spike a real regression or a promo the merchant ran? Does a decaying lift mean the novelty is wearing off, or that the test is simply converging? What should the merchant do? These questions don’t have clean answers. They need context, phrasing, and a recommendation. That’s the judgment part.
The shipped version fuses both jobs and hands the whole thing to Claude. The prompt says “check for these four patterns” (detection) and “write a human-readable message and a recommendation” (interpretation), in one call, over raw numbers. It gets detection right most of the time because the model is good and the series are small. But “most of the time,” on a task that has a right answer, is exactly the wrong place to spend a language model.
Why detection doesn’t belong in the model
The four patterns in that prompt aren’t exotic. Each one maps onto a piece of statistics that predates the transformer by decades:
- Sudden shift is changepoint detection. A rolling mean and standard deviation, or a CUSUM, will find a step change in a conversion series and tell you which day it happened, with a magnitude.
- Day-of-week pattern is seasonality. Group by weekday, compare the weekday means, and you have the effect size directly. No prose required.
- Novelty effect is a slope on the lift series: is the variant’s advantage over control declining over time? Fit a line to the daily lift, look at the sign and steepness of the slope.
- Trend divergence is the slope of the difference between the two series. Converging or diverging is a number, not a vibe.
None of these needs a language model to find. They need a language model exactly as much as computing an average does, which is to say not at all. And handing them to the model costs you three things you don’t get back:
Reproducibility. The same 21 days should always produce the same alerts. With the model in the detection path, they don’t, quite. Ask twice and the borderline cases flicker. A merchant who refreshes and sees an alert vanish has learned not to trust the panel.
Testability. I can write a unit test that plants a known changepoint on day 14 of a synthetic series and asserts the detector finds it on day 14. I cannot write a unit test that asserts “Claude noticed.” The deterministic version is the only one I can put behind a regression suite, which matters because this is a feature I want to change without breaking.
Recall on the tail. An LLM eyeballing a table can miss a genuine four-sigma spike buried in a noisy row, or invent a “slight upward trend” that a regression line would flatly deny. A z-score can’t miss the spike and won’t invent the trend. On the cases that matter most — the real, large, rare anomaly — the deterministic pass is more sensitive, not less.
The rebuild: detect in code, judge in Claude
So the version I’m building now inverts the flow. A deterministic pass runs first and computes the four signals over the daily series, emitting candidates with numbers attached:
type Candidate struct {
Type string // sudden_shift | day_pattern | novelty_effect | trend_divergence
Magnitude float64 // z-score, slope, or weekday delta — comparable within a type
Day string // the date it fires on, where that makes sense
Detail string // "control 3.1% weekday vs 2.4% weekend"
}
// candidates := detect(series) → []Candidate, filtered to |magnitude| over a real thresholdThe detect function is boring on purpose. A rolling z-score for sudden shifts, a weekday group-by for day-of-week, two least-squares slopes for novelty and divergence. Fifty lines of arithmetic, fully tested, no network call. The 10% threshold that used to live as a sentence in the prompt (“do NOT flag minor variation under 10%”) now lives in code, at the gate, where a threshold belongs.
Then, and only if there are candidates, Claude gets called. But the prompt is a completely different shape. It no longer says “find anomalies.” It says: here are the signals my detector already flagged, with their magnitudes — decide which ones are worth showing this merchant, and write the message and recommendation.
The statistical detector flagged these signals for experiment "Checkout CTA v2":
1. sudden_shift: variant B conversion dropped from 4.2% to 2.9% on 2026-07-09 (z = -3.4)
2. day_pattern: variant B converts 22% lower on weekends (weekday 4.1%, weekend 3.2%)
For each, decide if it is worth surfacing to the merchant and at what severity.
Consider whether signals explain each other (e.g. a shift that lands on a weekend
may be a day-of-week artifact, not a regression). Respond in JSON only: ...Now the model is doing the job it’s uniquely good at. It weighs the sudden shift against the day-of-week signal and can say “the drop on the 9th was a Thursday, so it isn’t the weekend pattern — this looks real.” It writes a sentence a merchant will actually read. It picks a severity. It recommends an action. None of that is statistics, and all of it is the part I couldn’t have written a rule for.
What the split actually buys
Every item on the “what’s missing” list from the original post falls out of this one change, which is why I’m convinced the split is the right seam and not just a tidier diagram:
Haiku, not Sonnet. The old prompt asked the model to reason over 21 days times N variants of raw data — a task where more reasoning genuinely helped, so it wanted Sonnet. The new prompt hands it a short structured summary of two or three pre-detected signals and asks for a verdict. That’s a narrow, well-shaped task, and it runs on Haiku 4.5 for a fraction of the cost, which matters when the volume grows.
A threshold that’s a threshold. In the original post I defended keeping the 10% in the prompt because a hard if in code “loses the model’s judgment” on borderline cases. I was half right. The honest version is: detection wants a hard threshold (reproducible, testable), and judgment on the borderline is a separate thing you can still hand the model. Now the code gate drops the clearly-trivial signals and Claude gets to make the call on the ones near the line — with the magnitude in front of it. Both parts get to be good at their job instead of splitting the difference.
Cheaper by construction. The model is called only when the deterministic pass finds a candidate. On a healthy experiment that’s converging cleanly, there are no candidates and there is no API call at all — the panel just says nothing’s wrong, correctly, for free. The old version paid for an inference every day per experiment whether or not there was anything to see.
This is the same spine, again
If you’ve read the other posts here, this shape should look familiar. The CRO audit eval uses exact-match scorers for the parts with right answers and LLM-as-judge only for the open-ended vision audit. The Upwork scoring pipeline puts a deterministic gate in front of the model and refuses to let Claude re-decide things a rule already settled. This is the third instance of the same principle in a different costume:
Do the part with a right answer in code. Hand the model the part that needs judgment, and hand it as little else as possible.
The failure mode is always the same too. It’s convenient to give the whole messy problem to the model, because the model will accept the whole messy problem and return something plausible. It takes deliberate effort to carve out the deterministic core and shrink the model’s job down to the irreducible judgment. But that carving is most of the engineering. The model is the easy part; deciding what not to ask it is the work.
Was the first version wrong to ship?
No — and I want to be clear about that, because the point of these posts isn’t to perform regret. The fused version was 100 lines, it worked, and it taught me exactly where the seam was. You often can’t see the right decomposition until you’ve shipped the wrong one and watched which parts flicker. Shipping the LLM-does-everything version first was correct. Rebuilding it once the volume and the cost and the flicker justified the work is also correct. The mistake would have been calling the first version done and never noticing that half of it was arithmetic in a trench coat.
The headline, if there is one: when a feature has “detection” or “classification” or “scoring” in its name, stop and ask which parts of it have right answers. Those parts are not LLM features. They’re the deterministic frame that makes the actual LLM feature — the judgment, the phrasing, the recommendation — small enough to get right.