Distribution accuracy measures — percentile

These accuracy measures can be used to evaluate how accurately a forecast distribution predicts a given actual value.

percentile_score(.dist, .actual, na.rm = TRUE, ...)

quantile_score(
  .dist,
  .actual,
  probs = c(0.05, 0.25, 0.5, 0.75, 0.95),
  na.rm = TRUE,
  ...
)

CRPS(.dist, .actual, n_quantiles = 1000, na.rm = TRUE, ...)

distribution_accuracy_measures

Format

An object of class list of length 2.

Arguments

.dist: The distribution of fitted values from the model, or forecasted values from the forecast.
.actual: A vector of responses matching the fitted values (for forecast accuracy, new_data must be provided).
na.rm: Remove the missing values before calculating the accuracy measure
...: Additional arguments for each measure.
probs: A vector of probabilities at which the metric is evaluated.
n_quantiles: The number of quantiles to use in approximating CRPS when an exact solution is not available.

Quantile/percentile score (pinball loss)

A quantile (or percentile) score evaluates how accurately a set of quantiles (or percentiles) from the distribution match the given actual value. This score uses a pinball loss function, and can be calculated via the average of the score function given below:

The score function $s_p(q_p,y)$ is given by $(1-p)(q_p-y)$ if $y < q_p$, and $p(y-q_p)$ if $y \ge q_p$. Where $p$ is the quantile probability, $q_p = F^{-1}(p)$ is the quantile with probability $p$, and $y$ is the actual value.

The resulting accuracy measure will average this score over all predicted points at all desired quantiles (defined via the probs argument).

The percentile score is uses the same method with probs set to all percentiles probs = seq(0.01, 0.99, 0.01).

Continuous ranked probability score (CRPS)

The continuous ranked probability score (CRPS) is the continuous analogue of the pinball loss quantile score defined above. Its value is twice the integral of the quantile score over all possible quantiles:

$$ CRPS(F,y) = 2 \int_0^1 s_p(q_p,y) dp $$

It can be computed directly from the distribution via:

$$ CRPS(F,y) = \int_{-\infty}^\infty (F(x) - 1{y\leq x})^2 dx $$

For some forecast distribution $F$ and actual value $y$.

Calculating the CRPS accuracy measure is computationally difficult for many distributions, however it can be computed quickly and exactly for Normal and emperical (sample) distributions. For other distributions the CRPS is approximated using the quantile score of many quantiles (using the number of quantiles specified in the n_quantiles argument).