How closed-loop calibration sharpens a producer's own engine.

A walkthrough of the ridge-regression calibration loop — what's being learned, from what data, how often, and what changes in a producer's rankings when it does. With a worked example from a real-shape 1,200-head composite operation in north-east Victoria.

The problem.

An industry breeding index is calibrated against the average producer in the breed's reference population. The coefficients are good — they reflect serious peer-reviewed work — but they reflect an average that doesn't exist on any specific farm. Two real composite operations selling weaners and yearlings into very different markets will see exactly the same EBV report and yet derive very different dollar value from the same animal.

The platform's first three engine layers — the bioeconomic weight derivation, discounted gene flow, and production-system modifiers — address this with closed-form derivation from the producer's profile. But they don't actually know what kills paid what on this producer's kill sheet last year. That's what the closed-loop calibration adds: an empirical layer that pulls the engine's weights toward what actually happened to this producer's animals at the kill floor.

What's being learned.

For each trait i in the producer's selection objective, the engine carries an industry-default weight w_i⁰. The closed-loop calibration loop fits a per-producer multiplier vector β̂ on top of that default. The actual weight used for rank-page derivation becomes:

The multiplier β̂_i is fit by ridge regression. The design matrix X is the per-animal EBV vector across the producer's slaughtered animals, and the target y is the per-animal net margin actually realised from each kill. Ridge regularisation handles the multicollinearity that inevitably exists between correlated breeding values (growth and milk, marbling and rib fat, etc.) without forcing the engine to pretend they're independent.

The choice of λ matters. Too small and the engine over-fits to recent kill noise; too large and the calibration loop is invisible and the producer's data is ignored. The platform uses leave-one-out cross-validation on the producer's own animals, with REML estimation of variance components when the kill set exceeds 50 animals. Below 50, the engine falls back to a published industry-prior λ⁰ with a smoothly-decaying weight toward the producer's own data as their kill set grows.

A worked example.

The numbers below are derived from a real-shape composite operation in north-east Victoria — 1,200 cows joined annually, weaner and yearling progeny sold across two grids (one MSA marbling grid, one feeder grid). Per-head net margin at each kill ranges from $920 to $1,450. The four-year kill set has 1,140 animals with full EBV records.

What the calibration loop reveals on this producer's data is interesting on its own:

• Growth EBVs are pulling more than the industry default suggests (β̂ for 200WT = 1.08, β̂ for 600WT = 1.22). The producer's feeder-grid contract premium for heavier yearlings is steeper than the breed-society average — and the realised premium shows up in the regression.
• IMF is pulling less than industry default (β̂ = 0.74). The reason is structural: this producer's MSA-grid premium peaks at marbling score 3 and then flattens — the industry-default IMF coefficient assumes a linear price curve, but on this producer's actual grid the curve plateaus, so marginal IMF beyond a threshold is worth less than the breed-society average.
• Milk is pulling much harder than industry default (β̂ = 1.34). This is a maternal-heavy composite operation with high retention — the discounted gene flow already valued milk highly, but the realised weight is higher still because daughters' milk EBV is showing up as a strong predictor of their progeny's slaughter outcomes too.
• Rib fat is pulling less negative than industry default (β̂ = 0.62). The producer's grid doesn't penalise excess rib fat as severely as the breed-society average, so the penalty applied at the rank page is correspondingly softer.

These per-trait multipliers don't replace the underlying bioeconomic derivation — they refine it. The closed-form derivation got the producer 80% of the way there from a 5-minute farm profile. The closed-loop calibration takes the remaining 20% from their own kill data, and the longer the producer uses the platform, the tighter that calibration gets.

What it looks like in the source.

The actual calibration code (simplified shape; the production source has additional logic for missing-data handling and bootstrap confidence intervals):

// core/js/calibration.js — fit per-producer β multiplier from kill data
async function fitProducerCalibration(producerId, traits) {
  // 1. Load realised kill outcomes for this producer
  const kills = await loadKillSet(producerId);
  if (kills.length < 50) return { beta: defaultPrior, source: 'industry-prior' };

  // 2. Build X (EBV design) and y (realised per-head net margin)
  const X = kills.map(k => traits.map(t => k.ebv[t.code] ?? 0));
  const y = kills.map(k => k.realisedNetMarginPerHead);

  // 3. Choose λ via leave-one-out cross-validation on this producer's data
  const lambda = chooseLambdaLOOCV(X, y, { lambdas: LAMBDA_GRID });

  // 4. Fit ridge regression: β̂ = (XᵀX + λI)⁻¹ Xᵀy
  const XtX = matMul(transpose(X), X);
  const regularised = addLambdaIdentity(XtX, lambda);
  const Xty = matVec(transpose(X), y);
  const beta = solve(regularised, Xty);

  // 5. Bootstrap a 90% CI on each multiplier for the rank-page tooltip
  const ci = bootstrapCI(X, y, { lambda, B: 1000, alpha: 0.10 });

  return { beta, ci, lambda, n: kills.length, source: 'producer-fit' };
}

Two design choices in this code that are worth flagging for the for-researchers audience:

• The fallback to industry prior at n < 50. Below that threshold, the regression is too unstable to be useful. The platform uses a smoothly weighted blend between the producer-fit and the industry prior across the 50–200 range so the producer never sees a sudden ranking shift when their kill set crosses a threshold.
• The bootstrap confidence interval on each multiplier. The rank page surfaces a per-trait confidence bar showing how settled each multiplier is. Producers can see at a glance which of their EBV weights are well-calibrated to their kill data and which are still dominated by the industry prior.

What changes when this loop runs.

Calibration runs nightly across the network. For a producer with a steady kill set, the per-trait multipliers shift only marginally from day to day — the change is dominated by new kill records ageing into the regression and old records ageing out. The interesting changes happen at structural moments:

• End of kill season. When a producer's late-season kills come in, the calibration shifts to reflect the seasonal grid mix. A producer who fills mostly feeder-grid contracts in spring and MSA-grid contracts in autumn will see their MILK and IMF multipliers move slightly each season.
• Grid contract change. When a producer signs a new processor contract with different price points, the next 12 months of kills under the new contract dominate the regression. The platform flags this in the rank-page tooltip so the producer can see which multipliers are about to shift.
• Drought / forced offload. Kills outside the producer's normal market timing carry less weight in the regression — the platform tags them as exceptional and down-weights them in the fitting step.

What the producer sees: a slightly different bull ranking each kill season. What the engine knows: the producer's calibration is sharpening, one kill at a time.

Why this matters for the platform's open claim.

Most commercial breeding indexes are calibrated and published once, then re-fit yearly or quarterly against the breed-society's reference population. The reference population is the producer body as a whole; the calibration cycle is institutional. Genemap inverts that — every producer's index is calibrated against their own animals every night, and the calibration code is in the open source tree, citable, reproducible, and overridable.

The Hazel (1943) selection-index framework was built on this idea: that the economic weights driving the index should be the producer's, not the industry's. For six decades the maths existed and the data didn't. The data exists now.