ref_select

Usage

ref_select(
  specs,
  data,
  outcomes,
  folds = 5,
  shape_threshold = 0.02,
  allow_uncalibrated = FALSE,
  ...
)

Every criterion is computed from out-of-fold cross-fitted scores:

Arguments

Argument	Description
specs	A named list of ref_spec objects, simplest first.
data	Reference data.
outcomes	Outcome selection.
folds	Cross-fit folds.
shape_threshold	Extra log-score gain required to accept a covariate-dependent skew/tail model.
allow_uncalibrated	If FALSE (the default), abort when no usable candidate passes every calibration gate. Set to TRUE only for an explicitly labelled exploratory comparison.
...	Passed to ref_crossfit().

Details

1. Drop failed or unstable fits.
2. Drop models whose out-of-fold calibration is unacceptable (the calibration gate below).
3. Rank survivors by out-of-fold mean log score; ties are broken by out-of-fold CRPS.
4. Keep the candidates within one standard error of the best, where the standard error is that of the paired per-observation log-density difference between the candidate and the best model (column se_log_score_paired; 0 for the best model itself). A candidate whose CRPS is worse than the best by more than one paired standard error (se_crps_paired) is dropped as well.
5. Require shape_threshold extra mean log score before accepting any spec with covariate-dependent skew or tail.
6. Choose the simplest remaining model (lowest ladder level).

The calibration gate passes a model when, for every outcome with $n$ out-of-fold scores, $|\bar z| \le 0.10 + 2/\sqrt{n}$, $0.80 - 2\sqrt{2/n} \le \mathrm{var}(z) \le 1.25 + 2\sqrt{2/n}$, the 95% coverage lies in $[0.93, 0.97] \pm 2\sqrt{0.05 \cdot 0.95/n}$, and each reported two-sided tail rate lies within its binomial tolerance. The gate also checks MACE, skewness, excess kurtosis, and material conditional location/scale drift. The fixed margins are practical tolerances; the $2/\sqrt{n}$ terms are two standard errors under perfect calibration, so small samples are not rejected for noise alone. The calibrated column of comparison records the gate’s verdict per model. If every usable candidate fails, selection aborts unless the caller explicitly opts into an uncalibrated result.

Value

A ref_selection: the selected spec and name, the selected model’s cross-fit scores, and a comparison table with the out-of-fold log score, CRPS, the paired standard errors used by the one-SE rule, and calibration gate values per model.