ref_assess

Usage

ref_assess(
  fit,
  newdata,
  by = NULL,
  uncertainty = c("total", "conditional"),
  allow_in_sample = FALSE,
  allow_calibration_reuse = FALSE
)

Reports overall probabilistic fit, marginal calibration, conditional calibration, and tail calibration on held-out data.

Arguments

Argument	Description
fit	A ref_fit.
newdata	Held-out validation data (required; training data are not stored on the fit). For out-of-fold evaluation use ref_crossfit().
by	Optional grouping column; the marginal calibration table is then reported per group (column .group).
uncertainty	Passed to predict.ref_fit(). "conditional" scores the point-estimate predictive; "total" (the default) integrates over coefficient uncertainty and matches predict.ref_fit().
allow_in_sample	Set to TRUE only for explicitly labelled training diagnostics. The default aborts if newdata exactly matches the fitting data; use ref_crossfit() for honest reference-sample assessment.
allow_calibration_reuse	Set to TRUE only for an explicitly labelled diagnostic on the same rows used to estimate a calibration map. The default requires evaluation data independent of fitting and calibration.

Details

overall maps onto the PCNtoolkit evaluation metrics as follows: standardized_log_score is the negative MSLL: the mean log score minus the log score of a Gaussian with the reference sample’s mean and population standard deviation (fit$reference_baseline, recorded by ref_fit()). The baseline never uses the held-out sample’s own moments.

Note that this is not the same baseline PCNtoolkit uses. PCNtoolkit fits its baseline Gaussian to the sample being scored, so its MSLL is computed against a held-out oracle and the two numbers differ by the log-score gap between the two baselines (beyond the sign convention). Compare against PCNtoolkit only after rescoring both sides on one explicitly chosen baseline.

cor is the Pearson correlation of observed and predicted median; PCNtoolkit’s Rho column is a Spearman correlation, so the two are related but not interchangeable. smse is the standardised mean squared error $\mathrm{MSE}/\mathrm{var}(y)$; ev is explained variance $1 - \mathrm{var}(y - \hat y)/\mathrm{var}(y)$. rmse, mae, mean_log_score, and crps are the usual proper and point scores.

marginal reports the moments of $z$ (mean_z, var_z, skew_z, excess_kurtosis_z), the Shapiro-Wilk statistic shapiro_w of $z$ (on a fixed-seed subsample of 5000 when $n$ is larger), the coverage of central intervals, and mace, PCNtoolkit’s mean absolute centile error: the mean over the centiles 0.05, 0.25, 0.5, 0.75, and 0.95 of the absolute difference between the centile and the fraction of observations at or below its curve.

conditional checks the scores against every covariate the model uses. For a numeric covariate it fits light diagnostic GAMs of $z$ and $z^2 - 1$ against it and reports the largest absolute fitted value with its pointwise standard error (location_drift, location_se, scale_drift, scale_se; descriptive amplitudes, since the maximum is selected) and the approximate p-value of the smooth term (location_p, scale_p), which is the test of no drift. For a factor covariate it reports one row per level with the mean of $z$ and the excess of $\mathrm{var}(z)$ over 1, their standard errors, and two-sided p-values.

Value

An object of class ref_assessment with overall, marginal, conditional, and tail tibbles plus the scores.